How Big Can AI Models Get?

ByteDance, the company behind TikTok, is really pushing the envelope with its latest AI model development endeavor - we're talking a model with up to 10 trillion parameters. This is a huge undertaking, and it's all part of the company's efforts to give Anthropic's Mythos system a run for its money and establish itself as a major player in the AI field. The model is still in the pre-training stage, and it's expected to take around three to six months to complete - after that, it'll be fine-tuned and potentially released.

The fact that ByteDance is developing such a large AI model is pretty significant, because it shows just how ambitious Chinese labs are getting - they're not just trying to catch up with their US peers, they're trying to outdo them. And it's not just talk - in recent weeks, Chinese models have been performing really strongly on benchmarks, with some of them coming close to Anthropic's Fable 5 in certain areas. ByteDance's approach is particularly notable, though, because they're building their models from scratch, rather than "distilling" existing ones from other labs.

This approach has been in place for over a year now, and it may have contributed to ByteDance's slower development compared to its rivals - but the company's management, led by founder Zhang Yiming, is convinced that it's the key to creating models that really outperform others. Zhang's all about targeting "world-leading model capabilities" in the long run, even if it means falling behind in the short term.

For people who work with AI, or just use it, the development of massive AI models like ByteDance's has some big implications. As these models get bigger and more complex, they can process and generate some really sophisticated data - which could lead to some major breakthroughs in areas like natural language processing, computer vision, and decision-making. But, of course, there are also some concerns about data quality, training methods, and security that need to be carefully addressed, to make sure AI is being used responsibly.

Source: Ars Technica