This post was originally published on this site.
(PeopleImages.com – Yuri A/Shutterstock)
The AI revolution is shaking up the tech world in ways both big and small. In addition to rewiring the data stack, AI is also changing the jobs that people do, and one of the biggest changes is the emergence of the AI engineer.
At first glance, you might think that an AI engineer is someone who builds AI applications. That’s partly true, but it’s not the whole story. Instead, an AI engineer is someone who uses AI tools to create applications.
“AI engineers are specialized professionals who use their knowledge of artificial intelligence and machine learning to develop computer applications and systems,” Coursera says in the description for its course, “What Is an AI Engineer (And How to Become One).”
As part of their jobs, AI engineers may work with generative AI tools and technologies, such as large language models (LLMs), other types of foundation models, vector databases, and prompting frameworks. They may be called upon to train a custom neural network for scratch, but most of the time, they’re using a pre-trained model. They also need to know how to assess the quality of data used in an AI applications, and how to access that data in real time to deliver an AI-powered outcome through whatever application they’re developing.
Chip Huyn’s book is the #1 best seller in machine theory on Amazon
The availability and accessibility of powerful foundation models is the big difference between data science activities of old and what today’s AI engineers are building, Chip Huyn writes in her O’Reilly book “AI Engineering.”
“If traditional ML engineering involves developing ML models, AI engineering leverages existing ones,” Huyn writes.
With existing foundation models as the core, AI engineers are using techniques like prompt engineering, retrieval-augmented generation (RAG), and fine-tuning to adapt foundation models to their needs, she says.
Four years ago, data engineers were taking over, says Eldad Farkash, the CEO and co-founder of Firebolt, which develops a real-time, cloud-based analytics database.
“Right now, we have AI engineers taking over,” he tells BigDATAwire. “What is an AI engineer? From what we’ve learned so far–and it’s all still in the making–it’s people that utilize AI to become much smarter than they were yesterday. It’s people that, with AI, can build production-grade solutions that they couldn’t be doing before.”
Data scientists and ML engineers can leverage their knowledge of math, machine learning frameworks, and a given industry or field to build custom data-driven solutions. But today’s AI engineers primarily are using off-the-shelf technology, such as pre-trained foundation models, to build entirely new applications that will disrupt industries, Farkash says.
The AI engineer requires expertise in software development, programming, data science, and data engineering, according to Microsoft. Though the job is similar in some respects to that of data engineer, AI engineers rarely have to write the code that develops scalable data sharing, it says.
“AI engineers are responsible for developing, programming and training the complex networks of algorithms that make up AI so that they can function like a human brain,” Microsoft says on its Learn site. “Instead, artificial intelligence developers locate and pull data from a variety of sources, create, develop and test machine learning models and then utilize application program interface (API) calls or embedded code to build and implement AI applications.”
GenAI is ripe to augment these functions (Source: “AI at Work: Why GenAI is More Likely to Support Workers Than Replace Them”)
The field of AI is changing quickly at the moment, and that’s impacting what skills you need to take advantage of the AI advances. AI copilots are already changing how software developers write applications, and the fields of data engineering, data management, and even data governance are also being impacted by AI. Alteryx recently published a survey that found data analyst jobs are more strategic now, thanks to the productivity boost provided to analysts by AI.
Prompt engineering and RAG skills–as well as knowledge of GraphRAG, which provides additional benefits–are important for today’s AI engineers. In the long term, AI engineers will play a large role in building and controlling the AI agents that companies are developing to automate decision-making. Rita Sallam, a distinguished VP analyst at Gartner, says AI engineers will work with knowledge engineers to create data and decision agents as part of AI-powered analytics.
While AI may do some of the work of existing engineers, the field of AI in the long run will drive the demand for more engineers, Gartner wrote in a research brief in October.
“Building AI-empowered software will demand a new breed of software professional, the AI engineer,” Philip Walsh, a senior principal analyst with Gartner, said in a press release. “The AI engineer possesses a unique combination of skills in software engineering, data science and AI/machine learning (ML), skills that are sought after.”
Demand for AI engineers is surging at the moment. The US Bureau of Labor Statistics says demand for AI engineers will go up by 23% by the end of the decade, exceeding demand for data scientists by one percent, according to a recent story in IEEE Spectrum. A Gartner survey from late 2023 found that 56% of UK and US respondents listed AI and ML engineers as the most in-demand role for 2024.
A year ago, the job board Indeed had 16,000 jobs listed for AI engineer. Today, the same search generates more than 27,000 jobs.
The surging demand translates to excellent pay. AI engineers make between $161,000 and $267,000 per year, with a mean of $206,000 per year, according to Glassdoor. That puts them in the higher echelons of IT jobs in terms of compensation.
As you search for the perfect AI engineer to take your GenAI applications to the next level, try to remain flexible with regard to your demands, advises Nicole Helmer, vice president of Skills and AI at Degreed.
“Companies have a long wish list of skills for AI engineers, which makes finding candidates who meet all of the desired requirements tough with such a breadth of complex concepts,” Helmer tells BigDATAwire via email. “Given how quickly the landscape is evolving, even if you hired the perfect candidate who met every requirement, within a year or two, if that person hasn’t been learning, they will no longer meet the expectations.”
Related Items:
How GenAI Will Impact Jobs In the Real World