The increasing adoption of AI technologies by New Zealand businesses raises novel challenges in relation to privacy and data use. AI systems rely heavily on the collection, analysis or generation of personal information in ways that are not always obvious to individuals.
As AI adoption accelerates, understanding how New Zealand privacy law applies in practice is becoming increasingly important.
In this second article in Bell Gully’s AI Series, we examine the key privacy risks across the AI lifecycle - from training data and inputs, processing and inference, to outputs and retention - and set out practical steps for managing them.
New Zealand privacy laws - where are the risks?
New Zealand has so far taken a relatively ‘light touch’ approach to AI regulation, preferring to rely on existing legal frameworks rather than introducing AI-specific rules1. As such, the Privacy Act 2020 and its Information Privacy Principles (IPPs) remain the core framework for privacy considerations related to AI.
While the Privacy Act does not refer specifically to AI, many of the IPPs have direct relevance to various aspects of the development and use of AI tools. In addition, where an AI system processes biometric information (for example, facial recognition technology) the recently introduced Biometric Processing Privacy Code 2025 imposes further obligations, including mandatory proportionality assessments before using biometrics and enhanced transparency requirements2.
Understanding where those risks are most likely to emerge, and which IPPs may be engaged at each stage in the AI development life cycle, can help organisations identify and manage privacy obligations more effectively. We consider some of the key risk areas below.
1. Collection of training data
AI systems are often trained using large datasets, which may include personal information collected from customers, employees or third-party providers. This is relevant to IPP 1 which requires personal information to be collected only where it is necessary for a lawful purpose connected with the agency’s functions or activities. AI development can create tension with this principle where organisations collect or retain large volumes of information on a speculative basis and on the assumption that the information may become useful later as AI capabilities evolve. Organisations should clearly define the intended use case before collecting or assembling training datasets and assess whether personal information is genuinely necessary for that purpose.
A related issue is disclosure. IPP 3 requires organisations collecting personal information directly from individuals to take reasonable steps to ensure those individuals are aware of certain matters such as the purpose of collection and intended recipients of the information. The new IPP 3A extends similar obligations to situations involving indirect collection. These principles are likely to become increasingly important in the AI context. Organisations should carefully review privacy notices to ensure they adequately address AI-related uses of personal information.
IPP 4 is also significant. It prohibits collection of personal information through unlawful means or through means that are unfair or unreasonably intrusive in the circumstances. This may arise where, for example, scraping activities capture sensitive information. In a recent example from overseas, a joint investigation by Canadian privacy commissioners found in May 2026 that OpenAI had engaged in “overly broad” data collection to train its initial ChatGPT models, collecting information without transparency or consent and launching ChatGPT “without having fully addressed known privacy issues”.3 Businesses should carefully review third-party providers’ terms and where possible seek warranties regarding the manner in which the information was collected.
2. Training and development of AI models
The process of training AI models can create privacy risks even where the underlying datasets are lawfully obtained. AI systems may retain or reproduce personal information, generate unexpected inferences, or create outputs that indirectly reveal sensitive details about individuals. There is also a risk that information initially collected for one purpose becomes embedded into systems used for entirely different purposes. These issues engage IPPs relating to the purpose of collection, fairness, security and use (IPPs 1, 4, 5 and 10).
To assist with managing those risks, businesses developing their own AI models should seek to exclude unnecessary personal information from training datasets and should test models to assess whether they can inadvertently reproduce personal information or other sensitive content in outputs. Where businesses rely on third-party AI providers, they should also review the applicable terms and configure relevant privacy settings carefully to ensure that personal information in any prompts, inputs and other data is not used to train the underlying model.
3. User prompts and inputs
Privacy risks can also arise from the way in which an AI tool is used in practice and the contents of input data. An organisation’s personnel may, for example, paste personal information into AI systems without understanding how the systems store or reuse inputs. Customer-facing AI tools may also collect personal information directly through prompts, chats or uploaded documents. These risks engage multiple IPPs including security obligations (IPP 5) and limits on disclosure (IPP 11).
To manage these risks organisations should provide staff training and implement clear policies governing the use of AI tools and restricting certain types of input data. In practice many businesses also adopt a tiered approach - prohibiting the use of public AI platforms for personal information or other sensitive information while permitting broader uses of data within certain approved tools.
4. AI-generated outputs and decisions
AI models can generate incorrect factual assertions or infer characteristics about individuals. That can create issues under IPP 8 which requires agencies to take reasonable steps to ensure personal information is accurate, up to date, complete, relevant and not misleading before using it. IPP 10 is also relevant, as it limits the use of personal information to the purpose for which it was originally collected, subject to limited exceptions. Questions may therefore arise where organisations seek to use existing datasets to support new AI-driven decision-making processes that differ materially from the original context of collection.
The Office of the Privacy Commissioner (OPC) has emphasised in its guidance that agencies remain responsible for decisions made using AI tools and should not assume that reliance on automated systems displaces existing Privacy Act obligations. In practice, businesses should ensure meaningful human oversight exists for significant decisions supported by AI systems and avoid over-reliance on automated outputs without independent review.
This issue has driven recent reforms overseas. From 10 December 2026, amendments to Australia’s privacy laws will require entities to disclose the use of “automated decision-making” where it could reasonably be expected to significantly affect an individual’s rights or interests. New Zealand may adopt similar amendments in future. The Law Commission has confirmed that later this year it will commence a review of legal issues related to the use of automated decision-making in the public sector.
5. Security and retention
AI systems can create complex security and data retention risks. Many AI tools retain prompts, outputs, usage logs and interaction histories by default, sometimes for extended periods and sometimes in overseas jurisdictions. Businesses should ensure they understand what information is being retained, where it is stored, who can access it, and whether it may later be reused to improve underlying AI models or related services. These issues can become particularly significant in the event of a privacy breach, where organisations may need to quickly determine what information was exposed for the purposes of mandatory notification obligations.
These risks engage several IPPs. IPP 5 requires agencies to ensure that personal information is protected by reasonable security safeguards against loss, unauthorised access, misuse or disclosure. AI systems can create challenges under IPP 5 where sensitive information is retained in ways not fully understood by the organisation deploying the tool. Where relying on third party vendors, businesses should carefully review the applicable contracts to ensure they include clear security obligations, including requirements relating to incident notification, retention and deletion of data, and appropriate restrictions on secondary use of personal information.
Key priorities for managing AI privacy risk
As AI technologies continue to evolve rapidly, businesses should reflect on whether their existing governance, compliance and risk management practices are sufficient to address the privacy risks associated with AI.
From an internal perspective, one of the most important steps is establishing clear governance around the adoption and use of AI tools. Organisations should develop and implement clear AI usage policies, practical staff training, and approval processes for higher-risk use cases involving personal information or automated decision-making. Privacy Impact Assessments (PIAs) are also likely to become more significant over time. While not mandatory, PIAs are strongly encouraged by the OPC as a useful means of identifying privacy risks and mitigants where personal information is being collected or used in connection with AI.
In relation to external risks and dealings with third-party AI providers, businesses should carefully review vendor contracts and privacy terms to understand how information is processed, retained and potentially reused. Where possible, this should include negotiating warranties regarding lawful collection practices, restrictions on secondary use of information, audit rights, incident notification obligations, deletion rights, and commitments around confidentiality and security standards. Where businesses have audit or information rights under vendor contracts, they should ensure that they exercise those rights in practice. This may include periodically reviewing security certifications, penetration testing summaries, retention settings, subcontractor arrangements and changes to model training practices.
Overseas, regulatory developments are evolving quickly. As noted above, Australia’s automated decision-making disclosure requirements commence in December 2026. The EU’s recently-enacted AI Act includes various data governance obligations, imposing strict controls on the use of personal data for training datasets. The OPC is clearly monitoring international developments closely. In February 2026, it joined more than 50 regulators worldwide in a joint statement expressing concern about the misuse of AI content generation systems, signalling increasing alignment with international regulatory counterparts4.
As regulatory expectations continue to evolve internationally, New Zealand businesses should monitor these developments closely and consider whether their governance frameworks, contractual arrangements and privacy statements are sufficiently future proofed to meet evolving standards.
If you have any questions about the matters raised in this article or require assistance with developing GenAI governance processes for your business, please get in touch with the contacts listed or your usual Bell Gully adviser.
In the next article in Bell Gully’s AI Series, we will be exploring the growing range of approaches to AI regulation in overseas jurisdictions and the key implications for New Zealand businesses.
1See our previous article NZ’s AI strategy: “light touch” regulation and opportunities for businesses - Bell Gully
2We discuss the Code in our separate article here.
3Statement by the Privacy Commissioner of Canada regarding a joint investigation of OpenAI’s ChatGPT
4Joint statement on AI Generated Imagery
Disclaimer: This publication is necessarily brief and general in nature. You should seek professional advice before taking any action in relation to the matters dealt with in this publication.
