The Importance of On Device AI for Developer Productivity

There has been a lot of speculation about what artificial intelligence (AI) will mean for software developers in the future, but one thing that’s clear is that AI-powered tools are already making a big impact on developer productivity. Pieces leverages on device AI to bring you real-time, context-aware suggestions from within your existing developer workflow. But why on-device AI? Let’s take a closer look.

What’s the Alternative?

Cloud-based AI is the alternative to on-device AI. As you can imagine, cloud-based AI is well positioned to handle distributed computing tasks or pull together data from multiple locations and process it centrally. In scenarios where you want AI to track or compare data across retail stores spread out across the country, for example, a distributed, cloud-based solution might be the best fit.

Cloud-based AI has several strengths that you see in other cloud-based scenarios, including more flexible scalability, ease of use, and cost effectiveness. Leveraging cloud-based AI tools for developers allows you to get up and running quickly without large investments in hardware and staffing to run your own data centers. It also enables you to scale up and down if you have fluctuations in demand, without investing additional resources or having your resources sit idle during times of low use.

However, if your goal is to get personalized learning, a highly secure system, and the fastest response times, then on-device AI is more efficient. On-device AI enables additional scenarios such as offline support, which can be important for high security work that is done on a local network.

On-Device AI Benefits

On-device AI, also known as Offline AI or Local LLMs (LLLMs), are AI systems that run on your local machine without requiring an internet connection. If your workflow is local and you want the most contextualized and secure experience with your artificial intelligence system, then using AI on device is your best choice.

Personalization

On device ML models (or AI) run locally and learn about you individually. Rather than taking data from multiple users and trying to generalize suggestions that will work for everyone, on-device AI works with the tools you use every day and optimizes for your personal workflow. It learns your environment, usage patterns, writing style, and more to create a personalized experience tailored to you.

For companies that isolate their staff from the internet for security reasons, you can use on prem AI solutions on a local server to enable better collaboration within your secure environment.

Performance

Not only will an AI copilot increase your team's efficiency, but on-device AI will get them the information they need even faster. Processing AI on-device avoids any latency that might be introduced by network traffic or distant servers. On device AI can also be run offline, offering flexibility and reliability during unplanned outages or in situations where the internet is not readily accessible, such as developing countries with unstable infrastructure.

Security

Another benefit to local processing is that it is inherently more secure. Any time you transfer or store data across multiple locations you increase the risk of data manipulation or loss. Keeping data out of the cloud reduces the risk of hacking and theft.

On device AI also keeps your data out of the cloud training models. Many of the cloud-based AI models, such as ChatGPT, leverage user input to train their LLMs. Downloaded LLMs don’t connect to the cloud, so your data is never shared or used in training. Pieces is SOC 2 Type II compliant, keeping you up to date with regulatory and compliance requirements that are especially important in fields such as healthcare, finance, and government.

Privacy

On-device AI opens up additional scenarios with regards to sensitive data. Since the data stays on your local machine, you don’t have to worry about exposing protected or personal information to the cloud and you still get the benefits of an AI copilot.

Cost

Running generative AI models in the cloud is increasing in cost and those costs are being passed down to consumers. Local models do not require cloud or network providers to run, which helps control the cost of on-device generative AI. Plus, when using local models, you are only consuming power from your individual device to generate a response which is much more environmentally friendly than cloud-models, which must ping large servers each prompt you create.

If you’re a developer wondering what the best LLM for coding is, you should definitely consider using local models like Mistral AI in your workflow.

Pieces On Device AI Copilot

Pieces works at the operating system-level to learn from interactions in your various development tools and your patterns of coding within them. We augment the existing tools that you use to research, code, and collaborate, so you have a tool-between-tools to reduce context switching, boost productivity, and provide centralized storage.

Pieces is built with small language models to automatically enrich code you save with useful context and metadata, along with other on device AI use cases we’ll discuss below. We took a local-first approach to storing and processing data, because we understand that speed, security, and privacy are all very important factors for developers trying to find the best AI code generation tools.

Personalization

As you can tell, Pieces is engineered to be hyper-personalized so it can speed up your development work. Pieces uses on-device machine learning models to create a centralized storage agent that works across your various developer tools in order to learn your end-to-end workflow and unify your developer experience. Pieces’ on-device AI offers the most personalization because it runs on your machine, connecting all of your existing developer tools and learning directly from your activity across the entire toolchain.

Tool Integrations

Pieces is integrated with some of the most popular developer tools, including Visual Studio Code, JetBrains IntelliJ, Google Chrome, Obsidian, and JupyterLab. You can go from researching and problem solving in the browser, to collaborating with teammates in Microsoft Teams, to coding in your IDE, without breaking your workflow and creating a rich knowledge base accessible to everyone on the team. Pieces manages your code snippets, automatically enriches them, and enables you to generate or explain any code using Pieces Copilot, without exposing your IP to the cloud.

Security

Pieces uses on-device AI to reduce online exposure by leveraging local LLMs like Phi-2, Llama2 and Mistral AI that you can download onto your machine. Local LLMs are also a good fit for companies that have more stringent compliance and privacy needs because they give companies greater control over their data. Learn more about the guidelines to secure AI from design to maintenance.

OCR

Pieces uses Optical Character Recognition (OCR) to convert handwritten or printed text from digital images like screenshots or scanned documents into machine-readable text. This technology significantly speeds up data retrieval tasks by allowing you to read screenshots and convert them to digital versions that can be easily searched. And because Pieces runs on-device, you can use its OCR to digitize sensitive documents locally with extremely fast compute power.

Pieces optimizes OCR through its own unique pre- and post-processing pipelines that work in conjunction with the Tesseract OCR engine. In particular, Pieces focuses on code snippets and their unique formatting to increase the reading input efficiency and fine-tune the output to recreate the proper structured syntax and formatting. For a deeper dive into OCR take a look at How We Made Our Optical Character Recognition Code More Accurate.

Automatic Enrichment

Pieces uses fine-tuned on-device machine learning models for auto-enrichment in less than 100 milliseconds. Small language models used on-device generate high-quality enrichments much faster than cloud models can. Using these distilled models through LoRA AI allows us to generate titles, descriptions, related links, tags, and other context for your code much faster and at a lower cost.

On Device AI Use Cases: The Future of Development

The future of AI is rapidly evolving and it’s important for companies of all sizes to explore how AI fits into their business so they don’t get left behind. Here are some areas we expect AI to have a big impact.

Better Communication

With Pieces, snippets of conversations can be saved in one location with rich context and tagging for future retrieval and cross-referencing. The wide array of Pieces plugins allow your developers to avoid context switching and stay on task while creating robust documentation.

Increased Efficiency

Pieces enables workflow integration by allowing developers to use the same AI copilot across tools that are already part of their workflow. You can share code or look up information from within the IDE, or leverage AI to generate or suggest code snippets as you’re working. AI helps automate manual tasks, freeing up developer time to focus on writing quality code. Plus, all of the code you save and conversations you have in one location are persisted across your workflow, creating a smarter, integrated AI experience.

Enhanced Security

Using on device and on-prem generative AI, Pieces provides air gapped security without sacrificing the power and potential generative AI brings to your enterprise. We are committed to maintaining the rigorous standards set forth in SOC 2 Type II and will continue to monitor the regulatory environment to ensure we stay up to date as the compliance landscape surrounding AI continues to evolve.