Data Engineering Archives - Page 2 of 2

How to spot a failing outsourced relationship

Posted on: 8 November 2021

A relationship breakdown is never easy, not least when it’s with your IT outsourcing partner. But what makes a seemingly good relationship go bad, and can you spot the signs of impending IT outsourcing failure before it’s too late? To get some insight from both sides of the relationship, we asked Backbone Connect Co-founder and Director, David McLeod, as well as our own CACI Network Services Sales Director, Liam Delaney, to share their outsourcing experiences, reveal the red flags to watch, and the secret to maintaining a successful relationship with an IT outsourcing partner. Here’s what they told us…

1. Communication has broken down

One of the earliest warning signs that your relationship with your IT outsourcing partner is flagging is that the frequency of your communication has dropped. “There’s always a honeymoon period with any new outsourcing relationship – the energy levels are high, and contact is constant,” explains David. “The issues arise when that contact becomes less routine and conversations turn forced and fractious,” he continues.

“Confusion about how a team should communicate with their outsourcing partner can also lead to protracted conversations and frustrations from both sides of the relationship if they’re not clearly defined at the outset,” says Liam. Further, changes over time can significantly contribute to communication barriers. “Through the duration of any long-term outsourcing relationship, team members leave, and a legacy starts to develop, which limits the potential of your outsourcing partnership,” says David. Liam agrees, “Whenever there’s a major personnel change on either side of the partnership, it’s time to review the service and make sure that it’s still meeting your needs.”

2. The vision has become (or already was) blurry

While both David and Liam agree that a successful IT outsourcing relationship is one that evolves over time, Liam highlights the necessity of starting the relationship with clear expectations. “You can’t outsource a problem that you can’t define,” he warns. “Outsourcing partnerships can bring a wealth of expertise and experience into your team as well as achieve cost savings, but you need to be clear on what success you’re looking to achieve.” If the goals aren’t clear, it can be difficult for an outsourcing provider to take effective action.

David also advocates working with outsourcing partners whose cultural values align with your business to ensure longevity in the relationship. “Your business’s culture is the one constant, unchangeable thing, so it should be one of the key measures you use when considering any potential outsourcing provider.” He adds, “Put simply, if you’re wearing t-shirts, and they arrive in business suits, you’re likely to have a problem.”

3. Fingers are being pointed

“When something goes wrong and blame is being thrown around, you stop being on the same team and your pathway forward becomes blocked,” says David. Liam agrees, “A good outsourcing provider is one that acts as an extension of your team, always looking to add value and deliver positive outcomes, especially when tackling an unexpected challenge.”

While it’s important to understand why a problem has occurred, both David and Liam agree that maintaining open, honest and constant communication can ensure both sides of an outsourcing relationship resolve conflicts and challenges together, although David notes that “when you seem to have a stream of issues, a stigma can become attached to the outside party, making it difficult for that partnership to continue effectively if it’s not addressed.”

Liam says that establishing a communications flow which facilitates continuous feedback is one way to avoid minor problems becoming bigger issues, although he also acknowledges the value in a proactive vendor – “At CACI, we’re always trying to anticipate our clients’ potential roadblocks and challenges, so we’re providing solutions before something becomes a problem.”

4. Your contract has become a constraint

A contract provides both parties in an outsourcing relationship the benefit of structure and protection, but it can become a barrier to progress when projects pivot in a new direction. Working with a vendor that can be flexible and offer an element of elasticity in their approach can help to avoid partners becoming stuck in a bind.

However, the size of an outsourcing provider can also impact on how agile a partner can afford to be, warns David. “Smaller organisations are typically more agile than bigger providers, but they can be highly volatile as they grow and evolve, which can lead to issues later. On the flip side, a very large outsourcing provider may not be able to offer the personal, value-add partnership that you’re looking for.”

Liam also advises that businesses pay attention to the finer details when firming up their outsourcing requirement. “It’s important to consider the unexpected and unusual use case scenarios. You can’t capture everything, but having awareness and alerting your vendor of the potential changes and challenges ahead means they can be prepared to act and adapt, preventing your project from coming to a standstill.”

5. You’re not growing together

“A clear sign that your outsourced relationship isn’t working is when you start to feel anchored,” says David. An outsourced relationship that continues to evolve and enhance your business as it grows is one that is truly valuable according to our experts. One way to form a relationship that adds long-term value is to select an outsourcing partner that has a wider capability offering. “I’m always thinking about the longevity of a relationship, looking beyond the initial requirement, and thinking about what else we can do to add value to our clients,” says Liam.

Nonetheless, capability isn’t the only thing to look out for. As Liam explains, having a future-focused mindset is also critical to a long-standing relationship. “I believe that the most successful partnerships are the ones where the provider brings both vision and value. They’re not just focused on what the client currently does, but they’re looking at what else they can be doing to improve.”

However, both our experts noted that, like any relationship, an outsourcing relationship requires investment and trust to realise its full potential. “It’s all about building and nurturing a partnership,” says Liam. David agrees and adds, “Trust is critical, and it’s not established overnight. Take the time to get the basics right – once you’ve got that with the right partner, you can achieve much bigger things.”

Looking for an outsourcing partner to help with your network operations? Contact our expert team today.

3 network transformation opportunities – and how to make them happen

Posted on: 28 September 2021

With digital transformation initiatives high on many organisations’ agendas and the impact of COVID-19 changing how most of us work forever, network transformation has never been more important.

And the truth is, there are a wide range of network transformation vendors to choose from. But most take a transactional approach to network transformation, delivering little value beyond the basic works carried out.

Rather than an “in and out” service, great network transformation relies on an end-to-end partnership-based approach, with your vendor working closely to understand all your requirements. It can also sometimes mean working across multiple lines of business and projects to deliver network transformation programmes at scale.

In working on multiple network transformation projects with one of our major transportation clients, there were some interesting opportunities we’d like to share. Here are three potential opportunities to be had by working with a network transformation specialist.

Opportunity #1 – Turn spare network capacity into additional revenue

For organisations with vast internal networks, there’s significant opportunity to commercially monetise spare capacity and offer greater flexibility to customers.

With a significant fibre network across its estate stretching tens of thousands of miles, our client realised that any spare capacity could be used for commercial applications to help generate additional revenue.

Using an end-to-end delivery process, we developed and launched a dark fibre service to help our client deliver connectivity as a service, in a way that was repeatable and efficient.

We kicked off the project by mapping out the client’s service lifecycle to identify any capability gaps. Once this was established, we brought together a selection of our client’s stakeholders virtually and ran interactive workshops to walk through draft processes, focusing on providing end customer service.

Following a successful service launch, several end customers now use our client’s services, with our client driving continuous improvements across the network. And we’re now working with our client to deploy the service across its wider network and develop a service model and approach for future deployments – allowing our client to develop its offering.

Opportunity #2 – Tackle customer complaints to strengthen relationships

In large organisations, it can often be a challenge to discover and resolve issues that directly impact customers. Following the regionalisation of our client’s legacy telecom assets, our client found that many of its asset managers were concerned about whether its legacy voice estate was fit for purpose.

Working with our client’s leadership team, we devised and managed a service improvement plan, which involved understanding the issues and what was needed to resolve them.

To help stakeholders mobilise the plan, we set up management reporting processes and acted as an intermediary between our client’s leadership and account management teams.

In just four weeks, we helped our client reach a resolution around its legacy voice estate, and moved the focus to other areas of concern – delivering improved service to our client’s regions.

Opportunity #3 – Dramatically reduce data centre costs

Outdated technology can be a significant drain on resources. And while it can be all too tempting to throw money and resources at the problem, this strategy can often cause more problems than it solves.

Our client had an ageing data centre infrastructure which was interfering with its ability to deliver a reliable service. What’s more, it was keen to ensure its mission-critical applications were always available. With its legacy technology approaching end of life and support, we recommended an infrastructure migration.

We worked closely with key stakeholders to create a test organisation at the start of the project to support the migration, helping our client build two new data centres with modern technology stacks.

Ensuring a thorough assurance process was used throughout to maintain regulatory compliance, we oversaw the design, implementation, and migration phases. And to confirm all programme deliverables were managed correctly, CACI developed a project management and testing platform using Jira and Zephyr.

By working closely with our client throughout the project, we helped the company save £40K and delivered the project two months ahead of schedule. Looking to the future, we’re now working with our client in an advisory role, helping it to shortlist an operating partner.

Network transformation: a world of opportunity

While network transformation opportunities can be wide-ranging and cover several lines of business, it’s important to select an outsourced vendor that also understands the importance of being a familiar point of contact.

Often embedded in clients’ teams for maximum impact, our end-to-end services allow our clients to benefit from a portfolio of skills and resources, helping them free up their teams to focus on more strategic activities.

To find out more about how we can help you design and implement network services – and even unite third party stakeholders – across your organisation, get in touch with our team of experts today.

7 signs that your company needs to outsource IT

Posted on: 21 September 2021

From reducing costs to meeting tight project deadlines and accessing specialist expertise, there are many advantages that come with outsourcing IT, but when does outsourcing offer the most benefit to businesses? We asked Brian Robertson, Resource Manager at CACI, to reveal the common signs that indicate a business would be better with an outsourced IT solution.

1. Your IT costs are high

Are budget worries keeping you up at night? Cost control is the most obvious reasons for businesses outsourcing IT. Indeed, a 2020 study by Whitelane Research found that 71% of UK businesses said that cost reduction was the main driver for outsourcing IT. But, is outsourcing really cost-effective?

“Just having a couple of IT specialists on your payroll can really rack up costs,” says Brian. It’s not just high salaries and the cost of employee benefits that are a concern. Companies that opt to run in-house IT departments also face the costs of purchasing, maintaining, and upgrading hardware as well as purchasing the software they need. “With outsourcing, these fixed costs become flexible, allowing you greater control of your budget,” says Brian.

2. You have skills gaps

The severe shortage in tech skills has long been a challenge for businesses, but as Brian explains, “The pandemic put organisations across every industry on a fast-speed trajectory to digitalisation.” He adds, “now, the focus is to keep that momentum going, but we’re seeing that many of our clients are looking for very specific expertise in a fiercely competitive and increasingly expensive marketplace.”

With recent research by ManpowerGroup finding that 69%, of employers globally are struggling to find workers with the right blend of technical and interpersonal skills, it’s clear that many businesses are fighting a losing battle. “This is where working with a trusted IT outsourcing partner can prove to be a strategic move,” says Brian. “A good outsourcer will always assess their client’s requirements holistically – matching skills and experience as well as cultural fit with end goals.”

3. Your IT infrastructure is outdated

“IT infrastructure is a vital component in every business, but it can become a huge drain on productivity, not to mention a growing security risk if not invested in,” warns Brian. He adds, “However, upgrading an outdated infrastructure is a resource investment that many lean I.T departments can ill-afford, creating a stalemate situation that prevents a business from maintaining competitive advantage.”

Therefore, if a business is struggling to maintain and manage its day-to-day IT operations, outsourcing may provide a practical solution. In addition to unlocking access to the latest and greatest tech, working with a reliable IT outsourcing partner will ensure your IT operations are optimised for enhanced performance, releasing your in-house staff to focus their efforts on achieving your business objectives.

4. Your business is vulnerable to security threats

Cyber security breaches are increasing. According to a survey released by GOV.UK last year, 46% of UK businesses and charities reported a cyber attack during the year, with 33% of those claiming they experienced a cyber breach at least once a week in 2020 – up from 22% in 2017.

The growing sophistication of cybercrime puts immense pressure on in-house teams as they struggle to stay on top of critical security practices such as 24/7 networking whilst also maintaining the myriad security systems they have in place. As Brian warns, “When it comes to cyber security, it’s not just a case of having the right technology in place, you need round-the-clock specialists that have the experience and expertise to utilise those tools and prevent potential threats before they become a problem.”

The global shortage in professionals with the right security skill sets are an additional challenge for businesses as they struggle to recruit and retain the specialists they need. Partnering with a trusted IT outsourcer can provide a cost-effective and reliable solution, as outsourcing removes vulnerabilities by ensuring a business’s security defences are ‘always on’.

5. Compliance is a concern

While cyber security is one concern, ensuring regulatory compliance is another, particularly in heavily regulated industries such as financial services. Failure to comply can lead to reputational damage and hefty fines, but to ensure compliance, organisations must have the capability to implement, maintain, monitor, and accurately report on IT infrastructure and security processes. As Brian explains, a partnership with a reliable IT outsourcer can offer significant value to a business that is under pressure to maintain compliance, “As well as providing the necessary resources and expertise to ensure compliance, an outsourcing partner will keep abreast of regulation changes, so your business is always one step ahead.”

6. You need flexibility

When you’re embarking on a new project, getting the right people with the right skill sets in place can be a difficult task. While upskilling your existing team members can be beneficial, inexperience coupled with a limited bandwidth can pose major risks to your project delivery as well as have a negative impact on your day-to-day operations. These problems are more acute if your delivery deadline is tight.

“Hiring new talent in-house is an option, but often it’s not the best one if a project is short-term or requires a range of specialist skill sets,” explains Brian. In these instances, partnering with an IT outsourcer can provide the most strategic, timely and cost-effective route forward because solutions are tailored to your specific needs. “Clients also gain from the insights and expertise of an experienced team – with the added benefit of elasticity to adapt if requirements change,” says Brian.

7. You need niche expertise

More budget-friendly than hiring a team of in-house specialists, and more reliable than challenging your existing team, outsourcing IT is often the most effective option when it comes to delivering projects that require niche expertise such as cyber security. Brian also highlights the benefit of introducing an outside perspective, “One of the most overlooked benefits of outsourcing is that businesses don’t just get access to specific skills and knowledge, they get to tap into a whole wealth of experience.”

“That’s why it’s so important to look for an IT outsourcing partner that has a proven record of proficiency and delivering results. Knowing what’s worked before, how to handle specific challenges, and what pitfalls to avoid –is truly invaluable to finding the solution that’s really going to work for your business.”

Looking for a reliable IT outsourcing partner? Share your requirements with our expert team today.

A sophisticated & extendable data platform for National Highways

Posted on: 7 July 2021

CACI Information Intelligence Group were delighted to be asked to help when National Highways needed assistance in the creation of a sophisticated and extendable data platform.

The Technical Standards Enterprise System (TSES) IT project was instigated from a recommendation accepted by the National Highways board that “the Design Manual for Roads and Bridges (DMRB) is developed and future proofed so that the investment arising from its review is fully leveraged, taking account of trends in digital technologies”.

THE CHALLENGE…

The challenge for the TSES was to facilitate the creation, maintenance and publication of a digitised version of all National Highways technical standards. The TSES will expose a digital DMRB to industry, promoting innovation and incorporating relevant systems. These included systems such as the Collaborative Authoring and Review System (CARS), the Departures Approval System (DAS) and a published online archive of all relevant DMRB documents.

THE SOLUTION…

The CACI team utilised open-source technologies to architect a sophisticated and extendable data platform for National Highways. The TSES is made up of multiple modules formed from common components that conform to open standards and frameworks. This approach has provided faster and more efficient development through re-use of components and enabled interoperability between modules, allowing National Highways to fully leverage the combined benefits of the TSES capabilities.

Two fully deployed modules are CARS and DAS.

CARS is an authoring and reviewing tool that allows teams that are geographically distributed and made up from National Highways supply chain, industry experts and devolved administrations such as Transport Scotland, the Northern Ireland Department for Infrastructure and the Welsh Government to efficiently write and review DMRB documents that adhere to the Manual for Document development rules. These rules apply specifically to the structure and verb forms that are used to ensure the quality of the standards.

DAS allows industry to apply for departures from a standard to upgrade safety features, efficiency in design of a road scheme or utilise advances in civil engineering practices.

The system is built using:

A microservice architecture and API service
PostgreSQL database
Microsoft Azure Cloud
Docker
NGINX
Spring Boot
Apache Tomcat
Angular Webapp
Keycloak

THE RESULTS…

The introduction of the TSES has dramatically increased the velocity and quality at which DMRB standards are being updated, enabling National Highways to meet their license mandate to finish the updating process by 2020. The digitisation of the documents has unlocked significant opportunities for realising the Digital Roads Vision including:

Facilitating a system-to-system link between the documents and CAD design systems.
Applying machine learning and artificial intelligence to enable augmented authoring of documents and tightening the feedback loop between departures and standards to ensure National Highways are able to benefit from engineering innovations.

CONTACT US

If you have any questions or want to learn more, get in touch today.

Data Engineering

Unite and integrate diverse data sets to uncover new insights and get more from your data.

The mitigation of unwanted bias in algorithms

Posted on: 14 January 2021

Unwanted Bias is prevalent in many current Machine Learning and Artificial Intelligence algorithms utilised by small and large enterprises alike. The reason for prefixing bias with “unwanted” is because bias is too often considered to be a bad thing in AI/ML, when in fact this is not always the case. Bias itself (without the negative implication) is what these algorithms rely on to do their job, otherwise what information could they use to categorise such data? But that does not mean all bias is equal.

Dangerous Reasoning

Comment sections throughout different articles and social media posts are plagued with people justifying the racial bias within ML/AI on light reflection and saliency. This dangerous reasoning can be explained for, perhaps, a very small percentage of basic computer vision programs out there but not frequently utilised ML/AI algorithms. The datasets utilised by these are created by humans, therefore prejudice in equals prejudice out. The data in, and training, thereafter, has a major part in creating bias. The justification doesn’t explain a multitude of other negative bias within algorithms, such as age and location bias within applying for a bank loan or gender bias in similar algorithms where it is also based on imagery.

Microsoft, Zoom, Twitter, and More

Tay

In March 2016, Microsoft released its brand-new Twitter AI, Tay. Within 16 hours after the launch, Tay was shut down.

Tay was designed to tweet similarly to that of a teenage American girl, and to learn new language and terms from the users of Twitter interacting with her. Within the 16 hours it was live, Tay went from being polite and pleased to meet everyone, to a total of over 96, 000 tweets of which most were reprehensible. These tweets ranged from anti-Semitic threats, racism and general death threats. Most of these tweets weren’t the AI’s own tweets and was just using a “repeat after me” feature implemented by Microsoft, which without a strong filter led to many of these abhorrent posts. Tay did also tweet some of her own “thoughts”, which were also offensive.

Tay demonstrates the need for a set of guidelines that should be followed, or a direct line of responsibility and ownership of issues that arise from the poor implementation of an AI/ML algorithm.

Tay was live for an extensive period, during this time many people saw and influenced Tay’s dictionary. Microsoft could have quickly paused tweets from Tay as soon as the bot’s functionality was abused.

Zoom & Twitter

Twitter user Colin Madland posted a tweet regarding an issue with Zoom cropping his colleagues head when using a virtual background. Zooms virtual background detection struggles to detect black faces in comparison to the accuracy when detecting a white face or objects that are closer to what it thinks is a white face, like the globe in the background in the second image.

After sharing his discovery, he then noticed that Twitter was cropping the image on most mobile previews to show his face over his colleagues, even after flipping the image. Amongst this discovery, people started testing a multitude of different examples, mainly gender and race-based examples. Twitters preview algorithm would choose to pick males over females, and white faces over black faces.

Exam Monitoring

Recently due to Coronavirus it has become more prevalent for institutions like universities to utilise face recognition for exam software, which aims to ensure you’re not cheating. Some consider it invasive and discriminatory, and recently it has caused controversy with poor recognition for people of colour.

To ensure ExamSoft’s test monitoring software doesn’t raise red flags, people were told to sit directly in front of a light source. With many facing this issue more often due to the current Coronavirus pandemic, this is yet another common hurdle that needs to be solved immediately in the realm of ML & AI.

Wrongfully Imprisoned

On 24th June 2020, the New York Times had reported on Robert Julian-Borchak Williams, who had been wrongfully imprisoned because of an algorithm. Mr Williams had received a call from the Detroit Police Department, which he initially believed to be a prank, However, just an hour later Mr Williams was arrested.

The felony warrant was for a theft committed at an upmarket store in Detroit, which Mr. Williams and his wife had checked out when it first opened.

This issue may be one of the first known accounts of wrongful conviction from a poorly made facial recognition match, but it certainly wasn’t the last.

Trustworthy AI According to the AI HLEG

There are three key factors that attribute to a trustworthy AI according to the AI HLEG (High-Level Expert Group on Artificial Intelligence – created by the EU Commission), these are:

It should be lawful, complying with all applicable laws and regulations;
It should be ethical, ensuring adherence to ethical principles and values; and
It should be robust, both from a technical and social perspective, since, even with good intentions, AI systems can cause unintentional harm.

These rules would need to be enforced throughout the algorithm’s lifecycle, due to different learning methods altering outputs that could potentially cause it to oppose these key factors. The timeframes where you evaluate the algorithm would ideally be deemed based on the volume of supervised and unsupervised learning the algorithm is undergoing on a specific timescale.

If you are creating a model, whether it’s to evaluate credit score or facial recognition, it’s trustworthiness should be evaluated. There are no current laws involving this maintenance and assurance – it is down to the company, or model owner, to assure lawfulness.

How Can a Company/Individual Combat This?

By following a pre-decided set of guidelines continuously and confidently, you can ensure that you, as a company/individual, are actively combatting unwanted bias. It is recommended to stay ahead of the curve in upcoming technology, whilst simultaneously thinking about potential issues with ethics and legality.

By using an algorithm with these shortfalls, you will inevitably repeat mistakes that have been already made. There are a few steps you can go through to ensure your algorithm doesn’t have the aforementioned bias’:

Train – your algorithm to the best of your ability with a reliant dataset.
Test – thoroughly to ensure there is no unwanted bias in the algorithm.
Assess – test results to figure out next steps that need to be done.

Companies that utilise algorithms, or even pioneering new tech, need to consider any potential new issues with ethics and legality, to assure no one is hurt ahead.

We can only see a short distance ahead, but we can see plenty there that needs to be done

A. Turing

Data analysis in a cyber security platform

Posted on: 15 July 2020

CACI Information Intelligence are working with the Government on a prototype cyber security platform to improve capability in the area of automated network defence.

The platform ingests messages from disparate sources, including mainstream cyber security sensors and several bespoke sensors that are unique to our customer, and analyse the resulting data so a corresponding action can be automated.

Before you can respond to a cyber threat you must first detect that threat in real time, with attackers aiming to subvert detection by any means necessary, sometimes ‘real time’ can mean data spread across days, months or years, depending on the value of the target.

Our ingest pipeline was designed with these factors in mind, developing a scalable solution using message queues and data stored in elastic search which allows us to receive a large amount of fine grained data from sensors. The data received into the system is characteristically small and noisy. The natural background noise on these networks can make it difficult to decipher a legitimate cyber-attack and sensors often only report small snippets of information. The first step is to normalise the data, extracting common features from the sensors data such as devices involved, files, URLs, timings and severity.

Once all data is normalised into a common model, we seek to understand more about it by passing it through an enrichment process. In an ideal world you would perform a high level of enrichment on every message, but this is computationally expensive – especially if it requires a 3rd party service such as a DNS lookup. We aim to perform a basic level of enrichment on every message, for example, by using internal databases we can geocode external IPs to their country. An asset library of all known devices within the system is a valuable resource, adding in device information, physical location and operational status. We can optionally use 3rd party enrichment or even query back into the network using tools like Osquery on an on-demand basis. This allows us to make decisions on how to enhance the dataset while balancing the load on the system and network.

Once the ingest process has finished enrichment, we have a large pool of data to analyse. To reduce the burden on the Cyber Analyst we make use of ML techniques. Using Recommendation Engines, we can look at the previous actions performed by the user for similar messages and make a suggestion on the correct response. If the confidence of the recommendation is high, then a response can be automated by the system, for example blocking an attacker’s access to the network.

Our UI is a key part of the application and allows for Cyber Analysts to browse the data within the system. They follow threads through the data, spotting patterns that could indicate the presence of an attacker. It’s important the UI can enable the user’s workflow of pivoting on the data and following these threads. We provide tools to group the data together to provide context. The system has built-in tasks to replicate these groups and present it to the user, should a similar pattern of events happen in the future. The sequence of events can be as important as the body of the events themselves, and our system’s UI accounts for this by enabling a timeline view.

Ultimately the automated data analysis and ML applied in this project means that the caseload of the Cyber Analyst is reduced, responses to cyber threats can be made at all hours of the day with significant levels of trust and can result in the fast, automatic removal of a malicious entity from an unmonitored network.