Untukmu.AI, an Indonesian gift recommendation platform, has implemented a privacy-protecting system using Meta's Llama artificial intelligence model, that processes sensitive customer data on users' devices, whilst still maintaining recommendation capabilities.
The company developed a semi-decentralised system using split inference, a technique that divides processing between user devices and cloud servers. The team selected Llama 3.1 8B for the project, citing its balance between output quality and resource efficiency, with the added benefit that it didn't require fine-tuning. The selection was further influenced by Llama's roadmap, which included potential future multimodal versions, including a rumored 405B parameter model.
The implementation splits Llama's 32 transformer layers across two checkpoints. The first layer processes data on the user's device, while the remaining 31 layers and output layer operate in the cloud.
The company's data visibility policy establishes three tiers of access:
- Users have full access to all their information and can monitor its use
- Untukmu.AI can access everything except customer personal data
- Third-party providers cannot see customers' personal data or the Untukmu.AI prompt, but can view their own prompt and the resulting output
The system can be adapted by third-party vendors, such as insurance companies or advertisers, to offer personalised products and services. With the end of the cookie era, and growing concerns about privacy and protecting Personally Identifiable Information (PII), Untukmu.AI's approach, suggests edge deployment is the way forward.