HomeB: Technical - Data and IT

Revision as of 13:18, 30 November 2015 by Joem (Talk | contribs)


IT, data handling, storage and analytics requirement for city-scale activity

This section considers the IT requirements of potential applications for smart energy data, ranging from premises-level up to whole-city interventions. We have divided potential data uses into the following four categories:

Managing demand

The motivations for managing demand may include saving money, energy and carbon, or increasing comfort. These ends may be achieved through absolute demand reduction, or by load-shifting (e.g. under a time-of-use tariff). Possible data inputs for demand management tools include:

  • Consumer Access Devices (CAD; see below) providing high-frequency demand data (~10 seconds) for use by visualisation / energy accounting tools
  • Smart meters enabling frequent and accurate billing (half-hourly)
  • In-building sub-meters supplying data to Building Energy Management systems (~1 second)
  • Home automation tools interacting with time-of-use tariff data to automate switching of loads to cheaper periods (~1 minute)
  • Non-intrusive load monitoring using very high-frequency (kHz, but not stored) sampling of electrical load to produce appliance switching timeseries for offline analysis

Demand management can take place at the level of individual premises; it can also be coordinated across multiple premises through automated demand response systems which aggregate the behaviour of a larger number of remotely switchable appliances. Early examples of this include "tele-switching" Economy 7 heating, in which an FM radio signal is broadcast to remotely switch storage heaters on and off. More recent examples include the use of TCP/IP network connections between large supermarket chains and National Grid, whereby refrigeration systems can be remotely switched off at times of high demand.

Managing the distribution network

Distribution Network Operators (DNOs) may develop an interest in operating automated demand systems across multiple buildings connected to certain points in the distribution network where headroom is limited. This is most likely to be associated with longer-term network asset planning, in which demand management interventions may offer cost (and carbon) effective alternatives to network reinforcement. It is also possible that DNOs will become involved in balancing supply and demand for electricity within the distribution network, performing a role currently handled centrally by National Grid. Data sources which could support these applications include:

  • Monitoring of demand at secondary substations; primary substations are already metered (~10s).
  • Realtime TCP/IP conversations with automated demand reduction systems in buildings on the network (similar to the existing arrangements between National Grid and supermarkets referred to above) - potentially in responses to signals from secondary substation meters.
  • Demand data from community-scale interventions in which CAD data is aggregated across a communities to support decision making around demand reduction vs network reinforcement (~30 minutes x 12 months x n buildings).

Managing supply

This application is concerned with maximising the value of power generation plant embedded in the distribution network within Bristol. Examples of such plant include several large wind turbines at Avonmouth, and a potentially large amount of building-mounted solar photovoltaics. Relevant data sources include:

  • Output metering from individual installations (~1s up to ~1 year).
  • Weather forecasting for output prediction (time and location).
  • Other info needed e.g. for power matching city [link?].

 Planning the local energy system 

Local energy system planning covers requires city-wide measurement and forecasting of heat and power demand over time and space, at granularities appropriate to specific tasks. The challenges here include:

  • Identifying opportunities for new low carbon infrastructure (e.g. heat distribution systems).
  • Managing the growth of existing systems.
  • Focusing infrastructure development into the optimal locations (e.g. preventing conflict between different technical approaches).
  • Identifying no-regrets infrastructure pathways to feed into investment and policy processes.
  • Targeting interventions to reduce fuel poverty, and feeding this into investment and policy processes.

The data requirements for these processes may include address-level half-hourly consumption of gas and electricity, and topoligically correct representations of heat, power and gas network infrastructure in the city. It is also likely to include detailed technical and socio-demographic descriptions of buildings and their occupants. Potential datasets include:

  •  Smart gas and electricity consumption data (halfhourly, address-level)
  • Substation monitoring electricity (~10s, OS grid refs)
  • Substation monitoring gas (?seconds, OS grid refs)
  • Embedded generation metering (30 minutes, OS grid refs, power network connection topology)
  • Input/output monitoring of CHP/DH (30 minutes, OS grid refs, gas and power network connection topology)
  • Weather measurement and forecasting for local supply and demand prediction (hourly, OS grid refs) 


Summarise applications, data, resolution, sources, actors and data volumes in a table, as per Data and IT Notes page

Produce a graph showing the potential data volumes for different applications, and commentary on what scales are likely to tractable (and which are not).

Write a section covering what we know, what we need to know, how we can find it out, about:

  • Access conditions for different datasets
  • Smart meter rollout plans (see below)
  • CADs, BEMs, NILM, and submetering (see below to an extent)
  • What's coming next, including IoT and automated DSR in devices (below?)
  • DNO metering and monitoring
  • GDO metering and monitoring
  • Local power generation metering and monitoring
  • Weather data availability and uses
  • Analytics for centralised coordination of automated DSR

NOTE: perhaps some of this can be discussed in the workshop on IT, comms and city energy planning?

Data comms and opportunities for data capture within anticipated flows

NOTE: I think this will come out of the above to do list - not necessarily a new section.

Smart meter functionality and consumer access device opportunities

Table of knowledge

Demand management (reduction, shifting) Supply value management  Scale: from building to city level
What do we know Feedback is essential to behaviour change.

Feedback at different granularities a1/2 hour to 10 second) and frequencies via smart meter is possible.

Disaggregation of end use is powerful

Between 3 end uses and all end uses can be disaggregated depending on data granularity.

The C HAN is accessible via the DCC - can send signals to smart enabled appliances in the home e.g on TOU.

 C HAN enables optimisation of local supply to local demand within the home.  HAN enables communications between all devices and local generation (pv on the roof) using zigbee protocol.
What do we need to know What frequency is data available to 3rd parties (e.g DNOs).

What tools are available for disaggregation and who will offer the disaggregation.

Where will it be hosted (in smart meter or in cloud).

When smart enabled appliances will begin to become available / if alternatives using plug devices will become available.

Functionality of home energy management systems.

    How aggregation can work at area levels - e.g how would an aggregator manipulate demand via smart meters to to meet grid needs.

Everyone in a area would have to sign up to the aggregator.

Where do we find it  Ask DECC to provide an accessible guide to SMETS2 functionality.      

SMETS2 compliant domestic smart meter functionality

Domestic smart meter functionality is still not set in stone, however the SMETS 2 functionality is largely agreed. The latest description of its technical functionality is found in DECC publication: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/381535/SMIP_E2E_SMETS2.pdf

According to Kate Thomas most of the remaining work on the specification is concerned with agreeing the protocols for the way that the DCC operates rather than in the fine detail of the functionality of the meters.

It is proving difficult to gain a concise picture of the functionality of SMETS 2 smart meters. So the list below will be incrementally added to as information becomes available:

  • Ability to respond to data requests from consumer access devices for electricity data every 10 seconds for gas and every half hour for electricity - Question: does this mean that energy consumption will be recorded over these periods or that the meter can respond to requests for data over these periods.
  • Smart Meters must support a minimum of 4 Consumer Access Devices one of which is the In-home display (which all consumers will be offered when they have a smart meter installed).

Data flows involving accessing smart meter data via the DCC are shown below:

accessing smart meter data via the DCC

With the consumer’s consent, an organisation will be able to retrieve data from smart meters using the communications infrastructure being managed by DCC. This route can provide access to half-hourly consumption and tariff information and will be available from the start of DCC Live Operations expected in 2016. To retrieve data via this route a business may either become a DCC User in its own right or they may enter into a contractual arrangement with a business that is a DCC User. DCC Users will need to demonstrate that they meet regulated privacy and security requirements on an on-going basis. A DCC User will be able to send an ad-hoc request to retrieve data from a meter, or to set a schedule with DCC to send regular requests to retrieve data (for example, monthly).

Ref: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/397291/2903086_DECC_cad_leaflet.pdf

Data flows involving linking a Consumer Access Device to the HAN are shown below:

connecting consumer devices to the Home Area Network

Accessing data via the Home Area Network

Smart Meters will establish a wireless ‘Home Area Network’ in a consumer’s home. This will be a local ZigBee wireless network (the SM HAN) which gas and electricity smart meters and in-home displays will use to exchange data. Consumers will also be able to pair other devices that operate the ZigBee Smart Energy Profile (SEP) to this network; such devices are typically known as Consumer Access Devices (CADs). The CADs being produced today are small boxes which connect to Wi-Fi routers to stream energy data, but the CADs of tomorrow could be anything from a tumble drier to a home automation hub. Smart Meters must support a minimum of 4 CADs; an In-home display (which all consumers will be offered when they have a smart meter installed) is a type of CAD. The processes by which consumers can pair CADs to the smart metering system are described in the box below. These processes will be available from the start of the main installation phase in 2016; separate requirements are in place for earlier installations.

Once a consumer has paired the device to their HAN, a CAD will be able to access updated consumption and tariff information directly from their smart meter; a CAD can request updates of electricity information every 10 seconds and gas information every 30 minutes. A device only needs to be paired once. Ref: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/397291/2903086_DECC_cad_leaflet.pdf

The Government has also committed to introducing a means of allowing consumers to pair a CAD without needing the involvement of a DCC User; a consumer will be able to initiate pairing of a CAD by using a function on their smart electricity meter. We are working with ZigBee on the detail of this process which we expect to be available at the earliest opportunity in the main installation stage which is expected to start in 2016.

Pairing a CAD

Pairing a CAD

The scope for consumer access devices

The trade association representing manufacturers of metering and control devices (BEAMA) have produced a guide to CADs.


The kinds of energy management functionality envisaged for a CAD connected to the smart meter HAN are summarised as:

  • Methods of measuring the primary energy input to the home (today by use of pulse and toroid type sensed meters).
  • Meters for the key energy using circuits (power and lighting), and sensors (zone thermostats and as required presence and lux sensors).
  • Controls (switches, dimmers, remote valve controls and circuit breakers).
  • A programmable energy manager module within the C HAN.

In addition to analysing the user’s incoming energy use (both primary energy from the grid and energy from onsite such as PV) it also measures the consumer’s energy by use type (heating, lighting etc) and manages energy use by:

  • Using sensors: temperature, light levels, absence etc coupled to programmed rules and controls.
  • Using programmed rules to ensure energy use remains within consumer’s maximum load target by implementing load shedding types of measures.
  • Where some form of TOU tariff is being used, applying limited load shifting by maximizing appropriate energy use at low tariff periods.

All of the functionality mentioned here happens outside of the SM HAN once the metering data is delivered across the CAD. The link to the availability of the CAD will significantly enhance the effectiveness of the energy manager module of a C HAN as:

  • Real time energy use and cost will be available to the CHAN enabling consumers to monitor accurately their energy inputs with that of heir energy use and cost by application in the home. This will enable them to take necessary action including making adjustments to certain programmed events.
  • Up to date and future TOU tariff data will be available to the C HAN to enable and provide users with cost optimised measures for their key energy uses.
  • Potential real-time peak shifting based for major energy uses where TOU tariffs justify this.

Privacy concerns and issues with responsibility for saving energy

Participants expressed a number of issues that concerned them. While some respondents were worried about being “bombarded with different energy suppliers” trying to compete for their custom (see also Balta-Ozkan et al., 2013a), others expressed concerns about their privacy. Indeed, the latter appears to be a common concern  in relation to smart services (e.g., Krishnamurti et al., 2012) as in both our own findings and in other qualitative studies participants have uttered the phrase “Big Brother” to invoke a comparison of being watched by an unseen and invasive presence (e.g., Balta-Ozkan et al., 2014; Fell et al., 2014).

Another concern that emerged was that the smart metering initiatives and their associated services would be used to place the responsibility for spiralling energy bills with householders. E.g., one of our focus group respondents commented, “See that worries me again, because again it feels like the onus keeps on being pushed back on the individual somehow that if you get this right you can reduce your energy consumption and somehow save money”.Indeed, in other studies scepticism about whether energy savings really could be achieved were a concern for participants (e.g., Forsa, 2010; Lineweber, 2011).  

Using smart meter and energy data to improve energy management – tools and apps

Table of knowledge

   Demand management (reduction, shifting) Supply value management  Scale: from building to city level
What do we know? Feedback is essential to behaviour change

Feedback at different granularities a1/2 hour to 10 second) and frequencies via smart meter is possible disaggregation of end use is powerful between 3 end uses and all end uses can be disaggregated depending on data granularity

The C HAN is accessible via the DCC - can send signals to smart enabled appliances in the home e.g on TOU

 C HAN enables optimisation of local supply to local demand within the home  HAN enables communications between all devices and local generation (pv on the roof) using zigbee protocol.
What do we need to know? What frequency is data available to 3rd parties (e.g DNOs)

What tools are available for disaggregation and who will offer the disaggregation. Where will it be hosted (in smart meter or in cloud) When smart enabled appliances will begin to become available / if alternatives using plug devices will become available

Functionality of home energy management systems

    How aggregation can work at area levels - e.g how would an aggregator manipulate demand via smart meters to to meet grid needs?

Everyone in an area would have to sign up to the aggregator.

Where do we find it?  Ask DECC to provide an accessible guide to SMETS2 functionality   

3rd parties accessing smart meter data to manage the grid

The organisations that operate the energy network infrastructure will be able to access data on an aggregated basis, to help them understand the loads on their network at the local level and to respond to loss of supply issues. They will have better information for managing and planning investment activities which will help the move towards ‘smart grids’ that allow the monitoring and active control of generation and demand in near real-time.

The potential for disaggregation

The amount of disaggregation into energy end use is driven by frequency of sampling of energy data. SMETS2 has a requirement that eletricity consumption data should be available in 10 second increments whereas gas consumption data should be available in half hourly increments. Both these data streams will be available in near real time via a Consumer Access Device. The level of disaggregation related to the sample frequency is shown below:

Disaggregation possible for different frequencies of sampling.jpg

Source http://web.stanford.edu/group/peec/cgi-bin/docs/behavior/research/disaggregation-armel.pdf.

This work which summarizes the findings of 40 papers on dis-aggregation finds that:

  • 1/2 hourly data can find 3 end use categories.
  • One minute to 1 second data allows for identification of 8 appliance types - this is the level of data that should be available through the smart meter.
  • Multiple khz (1000 readings per second) allows identification of 20-40 appliance types.
  • Mhz reading can identify individual lightbulbs.

Disaggregation tools

A paper describing the design of a free open source tool for comparing disaggregaton algorithms through non intrusive monitoring is shown here: http://arxiv.org/pdf/1404.3878v1.pdf.

The toolkit is available here: http://nilmtk.github.io/.

Some key questions

  • Can smart meter data be disaggregated into sufficient detail that inefficient appliances would be identifiable?
  • Can data be used to evaluate the effect of energy efficiency measures sufficiently well for business models to be developed around them i.e. Domestic ESCOs
  • Can consumers be reassured about privacy whilst sharing data with relevant actors to deliver the aforementioned benefits?

Integrating smart energy data and sensors and controls (‘internet of things’) – what’s in the pipeline and what benefits can it bring (energy and in other spheres – e.g. health)?

Table of knowledge

     Demand management (reduction, shifting) Supply value management  Scale: from building to city level
What do we know? information about energy consumption can tell us about behaviour patterns. In theory this can allow detection of unusual behaviour which could be used for safeguarding or telehealth

control of smart enabled devices remotely either via automation or manually offers potential for innovation in home security, safeguarding and lifestyle

vulnerable households are least likely to take advantage of smart home type technologies

What do we need to know?   Need to assemble a list of smart meter enabled services and opportunities

what level of disaggregation is needed to give robust indications of out of trend behaviours

What other sensors are available - e.g. co2, ultra Wide band, humidity, light levels

What are the new kinds of information that can be generated by synergistically bringing different kinds of sensor data together

Where do we find it?  EU projects including smart spaces and U smart consumer have assembled "landscape" reports of this kind of data        

Smart Home and Internet of Things

A smart home is a dwelling which incorporates a communications network that connects the key electrical appliances and services, and allows them to be remotely controlled, monitored or accessed. 

Many new homes are being built with the additional wiring and controls which are required to run advanced home automation systems.Retro-fitting (adding smart home technologies to an existing property) a houseto make it a smart home is obviously significantly more costly than adding the required technologies to a new home due to the complications of routing wires and placing sensors in appropriate places.

The range of different smart home technologies available is expanding rapidly along with developments in computer controls and sensors.This has inevitably led to compatibility issues and there is therefore a drive to standardise home automation technologies and protocols. In Europe, Installation Bus, or Instabus is becoming a recognised smart home technology protocol for digital communication between smart devices. Itconsists of a two-wire bus line that is installed along with normal electricalw iring. Instabus lines links appliances to a decentralised communication system and functions like a telephone line over which appliances can be controlled. The European Installation Bus Association is part of Konnex, an association that aims to standardise home and building networks in Europe.

Regardless of the technology, smart homes present some veryexciting opportunities to change the way we live and work, and to strengthen local generation and reduce energy consumption at the same time.

Exploratory activities for coping with Internet of Things are in the market with examples such as Whzan which is an internet based service for monitoring, controlling and optimising the use of a number of devices and systems from solar farms to telecare of the elderly and infirm.

Overview of the potential for Internet of Things systems to build a more sustainable and fairer energy system

This is the energy section of the "The Internet of Things: making the most of the Second Digital Revolution. a report by the governments chief scientific advisor" Ref: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/409774/14-1230-internet-of-things-review.pdf

The UK’s energy system is evolving. We are consuming less fossil fuel for heating, powering appliances, and transport. 14% of electricity was generated from renewable sources in 2013, a 300% increase from 10 years ago[1]. More than half of this came from intermittent sources such as wind power[2]. Solar photovoltaic generation, from rooftop micro-generation to multi-megawatt solar photovoltaic farms, is also growing in capacity. The UK’s energy infrastructure was built for a centralised model of generation, distribution and supply. It is now being stretched by the dynamic needs of energy suppliers and consumers. The Internet of Things could accelerate this trend. Econometric modelling commissioned by the Smart Grids Forum estimates that £27 billion of infrastructure investment is required to cope with forecast demand growth to 2050. Building a smart grid could reduce costs by 33% compared with traditional approaches[3]. Decentralised energy production and smart grids already exist in UK cities and on industrial estates. In rural areas, the emphasis is on adaptation of electricity distribution networks and innovative commercial arrangements to enable connections to onshore wind and solar photovoltaic farms as quickly and economically as possible. By 2025, digital technology may define the electricity system almost as much as the physical engineering. The Internet of Things could enable the transformation of our existing system, with far reaching impacts on consumers, suppliers and the infrastructure that connects them. However, the rapid intake of new technologies requires the right incentive structures for distribution network operators. The Internet of Things creates three major opportunities in the energy sector: reducing energy demand, managing patterns in demand and supply, and driving innovation.

Reducing energy demand

The £10.9 billion smart meters programme is the biggest government investment to date in Internet of Things technologies. By 2020, the national communications infrastructure will connect up to 53 million electricity and gas smart meters in homes and small businesses. A Home Area Network and linked in-home display will provide near real-time energy consumption information, and a central regulated body will control access to the data. Heating accounts for 79% of UK domestic energy use[4]. Evidence suggests that smart meters may initially provide 2-3% reduction in energy use through changes in behaviour[5]. However, many people will use energy savings in one area to enable themselves to achieve a higher level of comfort overall, instead of “banking” the financial and environmental benefits[6]. If meters are combined with thermostats, weather sensors and boilers, energy savings could range from 6-29%[7]. These technologies have the potential to reduce energy bills, carbon emissions and overall demand for electricity. There are other savings to be had in areas of domestic energy consumption. A marketplace of smart appliances that link to the Home Area Network, on which energy pricing data is available, would build on the benefits. An important opportunity presented by smart meters is helping to bring more people out of fuel poverty by supporting them in using energy efficiently, including by giving easier access to lower cost ‘time of use’ tariffs.

Managing energy patterns

Matching energy demand with supply is one of the biggest challenges for the energy sector over the next decade.61 More intermittent energy generation combined with a greater number of battery powered devices (such as electric vehicles and mobile devices) could exacerbate pressures in all parts of the system. Smart meters will give consumers and businesses better feedback and more control over their energy use, as well as providing electricity distribution companies with a more sophisticated means of managing the risk of an interruption to supply. Smart meters could help smooth out demand if supported by dynamic ‘time of use’ tariffs that reflect variable generation costs. The Internet of Things will also increase the potential for intelligent supply-side management. Smart grids will respond to changes in demand by balancing supply with storage and intermittent sources; and by maintaining supply to essential systems, even down to the level of individual devices. The use of battery storage in connected devices (such as electric vehicles) may be used to shift load away from peak demand. In the future, government may need to explore storage ncentives for households and small businesses, or to create additional flexibility by allowing mechanisations such as Distribution Network Operators to pay customers for providing storage capacity.

Driving innovation

The Internet of Things could create new business models for the provision of energy services. For example, new forms of demand management may lead to creative alternatives to traditional energy consumption patterns. In some parts of the USA, smart grids are being deployed to increase system resilience against ‘super storms’ that cause extensive disruption to electricity supplies. Richer and open data, alongside energy market reform, would create opportunities for new tariff models. Consumers may benefit from options that more closely reflect their individual energy demand patterns. Deploying Internet of Things technologies in energy is not without risk. To achieve the potential economic and societal benefits, three main threats must be managed: increased energy demand; insufficient security; and variable access.

Increased energy demand

The devices that enable integrated smart home services, such as heating, lighting, and electric vehicles will require energy to operate. Although the power drain of an individual sensor is likely to be minimal, in aggregate the Internet of Things may significantly increase electricity demand. This could be offset by changes to when the electricity is used, thereby reducing peak demand, but further research and modelling will be necessary to understand the implications.

Internet of things applications


In addition to the aforementioned, participants also noted some other ways in which smart meters could be used to benefit society, suggesting that altruistic as well as individual concerns may have some bearing on consumers responses to smart meters. For instance, one participant noted that if suppliers were able to disconnect electricity when they detected potential hazards then it might “potentially lower insurance premiums”, while another noted that smart meters could be used to “make sure old people’s homes don’t go below a certain temperature”. Some respondents also welcomed the suggested concepts from an “ecological point of view”, recognising that “there are other reasons other money

that you should switch it off”.

Engagement with DNO and GDO to establish real-time data feeds at various network nodes (cf just smart meters)

words here

Use of local data to minimise search costs to target assistance on those who need it most.

words here

  1. Renewable Energy in 2013, DECC
  2. http://www.renewableuk.com/en/news/press-releases.cfm/2014-07-31-new-uk-record-nearly-15-of-electricity-from-renewablesmore-than-half-from-wind
  3. ‘Smart Grid: a race worth winning?: a report on the economic benefits of smart grid’, Ernst and Young, 2012
  4. ‘The Future of Heating: A strategic framework for low carbon heat in the UK’, DECC, 2012
  5. ‘Smart Meter Rollout for the Domestic Sector (GB)’, DECC, 2011
  6. ‘The Rebound Effect: an assessment of the evidence for economy-wide energy savings from improved energy efficiency’, UKERC, 2007
  7. ‘Assessing the Use and Value of Energy Monitors in Great Britain’, BEAMA, 2014; ‘Hive Energy Saving Report’, British Gas, 2014