Critical insight

this report reveals the dual focus of artificial intelligence (AI) in data management: infrastructure construction and security governance. Although the investment of AI start-ups surged in 2024, the application of AI technology is still in its infancy.

Master to find

• insufficient AI preparation: about 70% of organizations are still testing new technologies. The primary task of IT leaders is to prepare for AI, but only 30% of respondents plan to increase IT budgets to support AI projects.   
• Data infrastructure construction: enterprises pay more attention to upgrading data storage and management technologies to build a technology stack suitable for AI. Data security and governance have become important concerns, especially in generative AI and unstructured data management.
• Technical and business challenges:
• technical challenges: data migration does not affect users and applications (54%), and uses AI for data classification (48%).
• Business challenges: preparing data for AI became the top challenge (57%), lacking visibility of storage expenditure (36%) and compliance with regulations (44%).
• Cost management and priority: cost Optimization (54%) is the top priority of data storage. IT teams face multiple priorities in response to modern disaster recovery and data cloud migration.
• Future skill requirements: nearly 60% of organizations need employees with data security, compliance, and sensitive data processing skills. Online training and recruitment are considered as major coping strategies.

Future direction

• enhance AI governance and security: as data security threats increase, organizations need to improve their data governance capabilities to ensure data accuracy and compliance with regulations.

• Build an AI ready infrastructure: IT leaders will continue to focus on building data infrastructure that supports AI and find a balance between internal development and cloud services.

Parameter test:

unstructured Data Management report 2023

unstructured Data Management report 2022

unstructured Data Management report 2021

summary

the global artificial intelligence boom is in the ascendant. According to PitchBook data, in the second quarter of 2024 alone, American artificial intelligence start-ups attracted $27.1 billion investment, accounting for nearly half of the total amount of US venture financing in the same period. However, this report reveals that enterprises are still in their infancy in practical application of these technologies.

Seventy percent of organizations are still testing new technologies, and "preparing for artificial intelligence" is still the top priority of IT leaders in the field of data storage and management. However, cost optimization has become more important this year, and organizations are trying to incorporate artificial intelligence into their existing IT budgets. Only 30% of respondents said they would increase the IT budget to support AI projects.

At the current stage of artificial intelligence development, IT departments have not imposed strict restrictions on data, users or tools. Enterprises focus more on building infrastructure and technology stacks suitable for artificial intelligence by upgrading data storage and management technologies. The study also shows that another key trend is to improve the security and governance of generative artificial intelligence and unstructured data management, while making up for the gap in security skills of IT talents.

As start-ups and mature technology suppliers compete to launch new artificial intelligence products and functions, enterprise IT teams are still laying the technical foundation and determining the economic strategies needed for startup. This year's survey shows that in the field of artificial intelligence, everything comes from data and how to manage, allocate, protect and prepare for new uses.

Key data

AI program

• only 13% of organizations restrict the use of enterprise data for artificial intelligence, while 31% have no restrictions on users, applications, or data.

• Nearly half (44%) are building AI-ready data infrastructure and 32% are developing their own learning models.

• Only 30% of organizations have increased the IT budget for AI projects.

• When preparing data for artificial intelligence, the primary challenge is management governance and security issues (45%), followed by data classification and tagging (41%).

• The main strategy to deal with AI data problems is to upgrade data storage and data management technologies (53%).

• the primary tasks of data storage in the coming year include cost optimization (54%), preparation for artificial intelligence (51%), and investment data management and data liquidity (41%).

• Projects that support artificial intelligence (30%) have a relatively low priority because IT organizations are facing competitive requirements for modern disaster recovery and backup (32%) and data cloud migration (38%).

• In terms of unstructured data, IT leaders focus most on providing data access for employees, followed by the convenience of data liquidity.

• migrating data without affecting users and applications has once again become a major technical challenge for unstructured data management (54%), followed by using artificial intelligence for data classification and segmentation (48%).

• Preparing for artificial intelligence top business challenges facing unstructured data management (57%).

• AI data governance and security primary future capability requirements for unstructured data management (47%).

• Nearly 60% of organizations need more employees with security, compliance, and sensitive data-related skills.

Data assets and expenditures

according to the survey, nearly half of enterprises store more than 5PB of unstructured data, and nearly 30% store more than 10PB. Storing and backing up billions of files in a hybrid cloud environment is costly. More than 55% of respondents said this year that this expenditure exceeded 30% of the IT budget. According to Gartner, the IT budget has increased by 8% this year, but funds are scattered in many demand areas, including preparing for artificial intelligence and recruiting more IT personnel. As in previous years, 90% of respondents plan to spend more on data storage this year than last year.

What is the amount of data you currently manage?

Storage and IT budget

Expenditure forecast for 2024

compared with 2023, will your spending on data storage increase in 2024?

Insight:

• the need to optimize storage costs is more urgent than ever.

• Enterprises need to balance data center costs with the complexity of cloud data and application management.

• AI Planning adds to this challenge because investing in AI-ready infrastructure and applications is costly.

• Storage-independent data lifecycle management is essential for properly placing data at appropriate storage levels to achieve cost savings and provides intelligent information about deleting duplicate or useless data.

Major challenges in unstructured data management

this year's survey requires participants to share technical and business challenges in managing unstructured data. Consistent with 2023, data migration without affecting users and applications remains the top technical challenge (54%). IT leaders are always committed to providing internal customers with excellent performance, avoiding conflicts and unnecessary technical support requests.

So, what is "influence"? It mainly refers to common problems when data is migrated or archived to new storage. Users are often confused and ask IT departments where their data is now. If the data is no longer in the original location or disconnected from another system, the application may fail. These problems stem from the proprietary methods used by some storage and traditional data management technologies in moving data.

The second biggest technical challenge is to use artificial intelligence to classify and Segment Data (48%). Data classification, that is, identifying and marking unstructured data, helps users search in a PB-level distributed environment. In the absence of a keyword-based approach to quickly find correct datasets, it becomes extremely difficult to use data for analysis and compliance management. Enhanced Data Classification is an emerging and efficient method, but best practices are still evolving.

Managing data lifecycle in different storage levels and cloud environments (45%) makes many organizations difficult. They lack deep visibility of data storage locations and their associated costs. In the cloud environment, services are constantly changing. This complexity partly explains the recent trend of migrating cloud workloads back to the local, but this may not be the best choice from the perspective of maximizing cost savings.

What are your major technical challenges in unstructured data management?

Unstructured number according to the main business service Challenge

since artificial intelligence involves complex security, privacy, ethics and legal issues in the commercial and social fields, preparing for artificial intelligence has become the primary business challenge for unstructured data management (57%). For more details about AI focus and policies, see the AI planning section below.

Lack of visibility to storage spending departments (36%) is the third major business challenge. This problem is particularly prominent under the background that enterprises implement the cost allocation policy of common responsibility to various departments.

Compliance with industry regulations and company policies (44%) is another hot topic. IT organizations not only have to deal with the high threat of ransomware, but also need to adapt to the increasing state and federal privacy regulations. These two threats are directly related to the way data is stored, shared, and accessed. Data security is no longer an isolated activity within an enterprise, but requires collaboration among multiple teams such as data storage and network professionals.

What are your major business challenges in unstructured data management?

Artificial intelligence meter row

although artificial intelligence has become a hot topic, there is no clear and successful plan to guide organizations to deploy artificial intelligence. As always, best practices depend on the needs, objectives, risk status and industry of the organization. At present, participants have adopted a relatively open attitude towards how employees use company data and artificial intelligence.

Only 13% of organizations restrict the data and artificial intelligence services available to employees. Most (60%) do not set any restrictions or only restrict the ability of some users to train artificial intelligence models.

More than 30% of organizations have no restrictions on artificial intelligence in terms of data, users, or applications. Although there are widespread concerns about the safety, ethics and accuracy of artificial intelligence, only 5% of organizations completely prohibit the use of artificial intelligence in the workplace.

How do you plan to combine artificial intelligence with your company's data?

Artificial intelligence capability policy

the process of supporting artificial intelligence involves many technical decisions, including choosing which large language models to use, deployment locations, required infrastructure, and how to train employees. IT leaders generally believe that building a data infrastructure suitable for artificial intelligence is crucial, and nearly half (44%) hold this view. Then, the trend of internal development of artificial intelligence models and the use of commercial services and/or cloud support for artificial intelligence is not clear enough. Although pre-built infrastructure, public models, and cloud services have advantages in terms of cost and ease of use, IT organizations are also weighing the retention of artificial intelligence internally to strengthen data governance, benefits of data transparency and data security.

How do you plan to use artificial intelligence?

Data concerns about generative artificial intelligence

since the release of ChatGPT in the autumn of 2022, negative cases of generative artificial intelligence have been common. Risks include: exposure of sensitive internal data to business language learning models that may be accessed by any external users, legal issues concerning the ownership of derivative works, the risk that private data such as personal identification information are accidentally imported into artificial intelligence, as well as false, misleading, and/or biased results due to lack of data transparency.

For generative AI, participants also worry about enterprise data leakage (27%) and data privacy and security violations (26%). Unpredictable AI costs (14%) ranked third. Considering the convenience of using generative artificial intelligence, only 4% of people worry that employees lack sufficient skills or knowledge.

What are the data management issues that you are most concerned about when using generative artificial intelligence?

Hole check:

the data risk of generative artificial intelligence is sudden, and many organizations are caught off guard when employees actively adopt it. In terms of artificial intelligence data governance, attention should be paid to five key areas: security, privacy, data sources, ownership and unstructured data governance. This article will discuss how to track data usage, discover and isolate sensitive data, and monitor data by enhancing visibility and search capabilities for all unstructured data.

AI budget

we asked participants how to pay for AI and the results showed that saving money in other areas was crucial. Only 30% of the respondents said they would increase the IT budget for artificial intelligence, while 34% would use existing cloud funds to pay. To some extent, it is reasonable to include artificial intelligence in the cloud budget, because major cloud service providers are investing heavily in artificial intelligence and machine learning technologies. On the other hand, in the post-epidemic era, enterprise IT organizations are facing stricter cloud spending reviews, and actual bills are often much higher than expected. The IT status study of Spiceworks in 2024 shows that only 3% of the software budget is allocated to artificial intelligence.

How do you budget for AI?

Insight:

cost optimization is the core theme of this year. By optimizing data management and storage, investing in energy-saving data center technologies, and taking strategic measures in cloud spending, organizations can find budget space for revenue-generating artificial intelligence projects. AI suppliers also need to be cost-effective-oriented to promote their products and services.

Major challenges in preparing data for AI

preparing artificial intelligence is the primary business challenge in unstructured data management, and data preparation is crucial in this process. Obviously, governance and security are the biggest challenges, with 45% recognition, followed by data classification and labeling (40%) and understanding of business requirements (38%). The vast majority of participants (87%) also said that the lack of appropriate tools posed challenges, mainly involving visibility, legacy technologies, and manual processes.

How to solve these challenges?

We also asked IT leaders how they would deal with these problems. They regarded modern data storage and data management platforms and the creation of internal working groups as the main solutions. This is followed by the strategy of using cloud-based artificial intelligence services, which can automatically create large language models, configure appropriate managed resources, and filter data. Artificial intelligence data workflow is a new concept. This technology can discover, classify, and automate the process of moving data to artificial intelligence tools, while enriching the metadata of unstructured data.

What are the major challenges you face when preparing data for AI tools?

How do you plan to address these challenges?

Data storage and unstructured data are preferred. Level

today, cost management has become the top priority for IT executives and executives responsible for data storage. Although hardware costs continue to decline and many types of Cloud Object Storage become more cost-effective, data cost management is still very complex. After all, most organizations are unwilling to delete any data out of concern that some data may be needed at some point in the future. In addition, backup and disaster recovery requirements lead to the generation of multiple data replicas, which are often stored on costly network attached storage devices.

What is your data storage priority in the next 12 months?

We also asked participants what their superiors were most concerned about. The answers mainly focus on meeting the needs of end users: data access, ease of use, performance and uptime.

What aspects of unstructured data do your IT leadership focus most on?

Insight:

preparing for artificial intelligence remains important, but it must not be at the expense of financial responsibility. IT organizations must first ensure that data is protected, available, and accessible in the most efficient manner. More strategic management of data can bring benefits in many aspects, including reasonable placement of data to achieve optimal economic benefits, enrichment and classification of data for artificial intelligence and machine learning, and ensure a good performance experience for end users.

Future capabilities of unstructured data management

data security threats continue to put pressure on IT professionals, and more protection layers are urgently needed to prevent hackers from invading systems that store enterprise data. Today, data storage technology usually has built-in security functions, so it is logical to expect unstructured data management software to support security, privacy, and governance requirements as well.

AI data governance not only protects data from leakage or abuse, but also ensures data accuracy, complies with industry regulations, manages data bias, and ensures that datasets do not lead to false, misleading, or defamatory results. The monitoring and alarm function (used to deal with capacity problems or abnormal situations) was selected as the first place last year and has attracted much attention this year. At the same time, the analysis and reporting function is also important.

Insight:

• they want to manage data risks more conveniently, for example, by providing a detailed search function to find personal identity information (PII) and automatically move it to a secure location according to enterprise policies.
• They want to know whether resources shared by users are abnormally active or whether the storage array is about to reach capacity.

Which unstructured data management capabilities do you think are the most important in the future?

IT skills and organization

this year, we added a part of the survey, focusing on IT skills and organizational change. The IT team faces the greatest challenge in filling the skills and knowledge gap in data security and compliance (58%). Understand that cloud storage is the second priority, followed by artificial intelligence policies and their business needs.

In contrast, a recent IDC survey found that North American IT leaders urgently need artificial intelligence skills, followed by IT operations and cloud skills, involving architecture, data management, and storage. IDC pointed out that these skill gaps led to missing revenue growth targets, quality problems and declining customer satisfaction.

What are the major skills and knowledge gaps you face in your data management/storage team?

Strategies to deal with skill gaps

facing the problem of skill gap, survey participants give priority to online training (62%), followed by recruitment and on-site training/activities (50%). This result can undoubtedly ease the worries of those who worry that artificial intelligence will replace their work, although 43% of participants rely on automation to fill this demand.

How do you solve these skills gaps?

Future IT organization reform plan

we also asked about the organizational changes that IT will implement in the coming year. The results showed that recruitment (63%) was the preferred strategy, followed by changes or restructuring of IT leadership (40%). The restructuring will include the adoption of a shared service model, with 31% of participants saying that this is a popular strategy that can reduce costs by simplifying the procurement, delivery, evaluation and billing of it services. The coming year will be a period of change for IT. The team needs to manage numerous competitive and changing priorities with limited resources.

What organizational changes will IT face in the coming year?

Five key points

1. Cost optimization is still a top priority

for the fourth consecutive year, the IT director said that the storage expenditure of that year would exceed that of the previous year. This year, the primary task of data storage is cost optimization, not preparation for artificial intelligence. Respondents said that their IT leaders are most concerned about ensuring that data is accessible, highly available, and easy to migrate when needed. They hope to achieve these goals without affecting users and applications, which is also the top technical challenge for unstructured data management this year.

2. Ai faces challenges

preparing for artificial intelligence involves a range of potential expenditures, including upgrading storage and computing infrastructure, cleaning and preparing data, training or developing customized language models, acquiring new IT skills, and enhancing security tools. IT will be very difficult to pay these fees, with only 30% of respondents saying they will increase the IT budget to support artificial intelligence. Most strategies will involve allocating funds from the existing budget. For example, 34% of respondents said they would squeeze funds from the cloud budget to fund artificial intelligence. It is essential to optimize data management to save costs, invest in energy-saving data center technologies, and refine cloud spending to avoid waste.

3. Security and governance issues hinder the development of artificial intelligence

respondents were equally concerned about corporate data leakage and privacy violations. The IT team needs at least to formulate policies and tools to prevent sensitive data from leaking into common language models and to audit the use of artificial intelligence by employees to reduce the incidence of false or misleading results. IT will modernize the data storage and management platform and create an internal working group as an initial step in the secure deployment of artificial intelligence.

4. Unstructured data management evolves for artificial intelligence governance

unstructured data management solutions are maturing, providing IT users with a way to easily migrate and tier new data storage and analyze costs. The IT team now hopes to have features that support Ai data governance and security, such as quickly finding, marking, and classifying sensitive data, and automatically moving IT to secure storage based on policies, to avoid being absorbed by generative AI. Other capabilities include creating artificial intelligence data workflows and integrating data management tools with artificial intelligence tools to find and mark sensitive datasets, such as personal identity information (PII), in large data environments.

5. Ai strategy focuses on infrastructure

at present, the core goal of IT leaders is to build an artificial intelligence ready data infrastructure ". There are several feasible ways to achieve this goal, including purchasing external technologies, independently developing internal Technologies, utilizing cloud services, or integrating the above two strategies to form a hybrid model. The IT team has different opinions on whether to independently develop artificial intelligence models or to use commercial services and/or cloud services. When making relevant decisions on artificial intelligence technology, internal professional knowledge accumulation and resource status must be comprehensively considered to support the implementation of new technologies, while considering key issues such as budget allocation and data security.

---[This article is finished]] ---

this article is reprinted from: Andy730 public account