Visit adept.ai → On Hacker News Sep 2022 – Oct 2023
AI & Machine Learning 4Systems & Performance 1Software Engineering 1Hardware & Electronics 1Tech Industry & Business 1
AI has moved at an incredible pace in the last few years. Scaling up Transformers has led to remarkable capabilities in language (e.g., GPT-3, PaLM, Chinchilla), code (e.g., Codex, AlphaCode), and image generation (e.g., DALL-E, Imagen).
We’re open-sourcing Fuyu-8B - a small version of the multimodal model that powers our product.
We’re open-sourcing Persimmon-8B, the most powerful fully permissively-licensed language model with <10 billion parameters.
Transformers have grown deeper and wider, but training them on long sequences remains difficult. The attention layer at their heart is the compute and memory bottleneck: doubling the sequence length would quadruple the runtime and memory requirements.