InftyAI @InftyAI

We just released v0.2.0 of AlphaTrion, an observability platform for GenAI applications. In the new release, we implemented features like new dashboard, token/model tracing, experiment management, multi-tenant support and more. See github.com/InftyAI/alphat…

Happy to announce that we just released v0.1.0 for AlphaTrion, an open-source framework to help build GenAI applications, like experiment tracking. Still in early stage, looking for talent people to help us. 🎉github.com/InftyAI/alphat…

More and more frameworks out there to build GenAI applications, especially for observability and evaluation: github.com/InftyAI/Awesom…

Welcome @yuchenrcheng to join our community! Thanks for building our website! 🎉

Join our discord server discord.gg/UWnjUG6X8j to explore more ! 👏

Glad to share the keynote about llmaz at KubeCon HK, we still have a long journey ahead. BTW, the food here is great!

keyboard_arrow_left Previous keyboard_arrow_right Next

👀 Large Model Inference on Hetergeneous chips using vLLM, by @kerthcet (@daocloud_io) challenges for LLM inference: heterogeneous clusters: - variety of chips - high-end GPU sold out - perf/cost tradeoffs AI gateway: needs to be aware of - LLM-wise routing, KVCache, LoRA load Advanced orchestration: - multi-host inference for DeepSeek R1 - xPyD paradigm - LLM specific autoscaling others: - fairness between workloads - efficient model loading - steep learning curve due to complexity

keyboard_arrow_left Previous keyboard_arrow_right Next

LLM serving + GPU spot instances is a good combo I believe, both for cost and GPU diversity, but we may have several hurdles ahead. See our proposal with llmaz here: github.com/InftyAI/llmaz/…

The scope of inference is vast and rapidly expanding. See awesome-llmops.inftyai.com. #llmops #inference #aitools

Join @kerthcet together to explore how to deploy vLLM with llmaz and more.

CNCF @CloudNativeFdn

a year ago

Kante Yin of DaoCloud takes the keynote stage at #KubeCon + #CloudNativeCon China! His talk covers adapting vLLM for Chinese GPUs and deploying it on Kubernetes using llmaz for heterogeneous scheduling. Session info: hubs.la/Q03m4Yg_0 Register now: hubs.la/Q03m4Z1K0

2 1 11 3K 0

TensorRT-LLM in-tree support with llmaz: github.com/InftyAI/llmaz/…. Thanks @Se7en258 🎉

Our website for llmaz is live now: llmaz.inftyai.com, thanks @yuchenrcheng for the foundation work. Join us to help optimize the page together.

Check awesome-llmops.inftyai.com to find your projects there. And extremely grateful to everyone in the community! 🙏 #llmops #aiinfrastructure #awesome

DeepWiki with llmaz😃: deepwiki.com/InftyAI/llmaz

repo: github.com/InftyAI/llmaz

If you ignore the Y-axis, it seems we're in a good shape!😂 We're planning to apply for the sandbox of CNCF to involve more contributors, more ideas, more suggestions. If you find this project a bit helpful, please drop us a star🌟, thanks! #OpenSource #cncf

llmaz just released v0.1.3 to employ Envoy AI Gateway from @EnvoyProxy as the default inference service entrance, thanks for @xu_paco efforts. We'll also work with the Envoy AI Gateway team in the future to push this project to the next level. Take a try github.com/InftyAI/llmaz/….

Nice work from @Se7en258 for automating the Project Request with issue tickets only. 🥹 Join us in maintaining the Awesome-LLMOps project and drop us a star if you find helpful to you, thanks! github.com/InftyAI/Awesom… #awesome #llmops #opensource

Thanks @kerthcet to be our first backer. We guarantee that all the funds will be used within our community openly and transparently! 🎉 Support us & Join us! opencollective.com/inftyai #InftyAI #OpenSource

llmaz joined the vllm community as one of the external integrations🎉. We're now focused on our v0.2.0 milestone which will bring the functionalities around lora and prefix cache managements. docs.vllm.ai/en/latest/depl…