Methodology – Mobile Accesses

Target Populationhttps://www.anacom.pt/render.jsp?contentId=55129
 
Sample Definition and Selectionhttps://www.anacom.pt/render.jsp?contentId=55130
 
Quality control of resultshttps://www.anacom.pt/render.jsp?contentId=55131
 
Precision of resultshttps://www.anacom.pt/render.jsp?contentId=55132
 
Weightinghttps://www.anacom.pt/render.jsp?contentId=55133
 
Technical Architecture, Operating Mode and Component Detailhttps://www.anacom.pt/render.jsp?contentId=55134

---

Target Population

The target population is made up of indoor and outdoor locations in the municipalities of Faro, Lisbon and Porto. Indoor locations are deemed to be private households, while outdoor locations comprise public locations such as shopping and leisure areas, airports, and schools, where there is a high usage of mobile broadband Internet.

Each member of the panel was tested for the three mobile service operators - Optimus, TMN and Vodafone. The criterion used for the recruitment of the test panel was based on the existence of proper UMTS coverage for the offerings being analysed.

Sample Definition and Selection

A longitudinal sample with rotation was used in the municipalities of Lisbon and Porto, where the study had already been carried out in 2008. A contact base was thus set up with the members from these municipalities which participated in the previous study and which showed interest in participating in the same study again. This methodology was applied without prejudice to the sample distribution defined for the overall measurements, regarding the distribution of population in each parish. This option is particularly suitable in this context because it leads to much better precision in the study of the evolution versus what would be achieved by using independent samples.

To feed the part of the sample that was not covered by the 2008 test panel, and for the municipality of Faro, a three-stage selection method was used for the indoor accesses.  The first phase involved the selection of the parishes to be included in the sample. Following that, the number of points to be observed in each parish was determined, and these were distributed in proportion to the population of each parish. Afterwards, a first sample of locations (corresponding to the first observations) was randomly selected from a contact base of the households of each municipality. Then the actual observations points were selected using the snowball method, thus guaranteeing that no more than one point was selected in the case of streets, squares, etc., and no more than two in the case of avenues. Recruiting was done by phone.

The outdoor accesses (in the municipality of Faro and in the incomplete sample from the 2008 panel in the municipalities of Lisbon and Porto) were selected from a contact base of these locations in each municipality with a view to ensuring geographical dispersion between the evaluated points and variety in the type of location (shopping centres and other leisure areas, universities, schools, airports, other).

The following maps present the actual observation points.

Figure 1 – Observations points in the municipality of Faro

(Click to enlarge image)

Figure 2 – Observations points in the municipality of Lisbon

(Click to enlarge image)

Figure 3 – Observations points in the municipality of Porto

(Click to enlarge image)

Table 1 shows the sample sizes obtained. It should be noted that a total of 61 measurements were carried out in each location of the panel: 9 for each day of the week and 8 for each day of the weekend.

Table 1 – Sample size by segment

Segment	Sample
UMTS users	540
UMTS users per Operator	180
Measurements: per indicator and operator	32,94
Measurements: per indicator, operator and city	10,98

We can see from the above table that a panel of 180 locations was set up, evenly spread between the municipalities of Lisbon, Faro and Porto (60 users per municipality). The three operators were evaluated at each location, which made it possible to set up a total sample of 540 users.

Regarding measurements per indicator and operator, at least three tests were carried out for each time period (one test for the FCCN server and two tests for international ISP servers, one of them in Europe and the other in the USA), meaning 3 tests X 61 periods X 180 households. This yielded 32,940 measurements per indicator and operator, and 10,980 measurements per indicator, operator and city (3 tests X 61 periods X 60 households).

The study focused on the commercial offerings of each operator, as follows.

Table 2 – Commercial offering under study per mobile access operator¹

Mobile operator	Technology	Commercial offering trade name	Maximum Download speed (Mbps)	Maximum Upload speed (Kbps)
Optimus	UMTA-HSDPA2	Kanguru Basic	2	384
TMN	UMTS-HSDPA	10/25 Pre-paid Broadband	2	384
Vodafone	UMTS-HSDPA	Vita Net Plus	2	384

Tests were carried out on weekdays and at weekends between 27 November 2009 and 7 February 2010³, according to the following schedule:

Table 3 – Distribution of mobile access tests

Order	Week days	Weekends
From 8h00 to 10h00	X	X
From 10h00 to 12h00	X	X
From 12h00 to 16h00	X	X
From 16h00 to 18h00	X	X
From 18h00 to 20h00	X	X
From 20h00 to 21h00	X	X
From 21h00 to 23h00	X	X
From 23h00 to 1h00	X	X
From 1h00 to 4h00	X	X

 
Quality control of results

Rigorous quality control was ensured for the work carried out during all stages. During the information collection stage, data collected was regularly monitored to detect any anomalies that could affect the quality of measurements. Later, quality control of the information collected included the analysis of atypical values. Outliers were identified on several measurements, explained by the servers’ downtime. These values were excluded.

Precision of results

The precision of the results⁴ obtained for an average, with a 95% confidence level, for one of the collected indicators (download speed, in this case) is described in the table below:

Table 4 – (Absolute and relative) precision for the download transfer speed, per operator, server and municipality

		Absolute Precision  (kbps)			Relative Precision
		OPTIMUS	TMN	Vodafone	OPTIMUS	TMN	Vodafone
FARO	USA	5.05	2.44	4.86	0.5%	0.3%	0.5%
	UK	17.49	5.75	16.66	1.1%	0.5%	1.0%
	National	19.77	7.27	19.90	1.3%	0.6%	1.2%
LISBON	USA	4.61	2.15	5.17	0.4%	0.3%	0.5%
	UK	18.19	5.82	17.15	1.0%	0.5%	0.9%
	National	19.36	7.00	19.09	1.0%	0.5%	1.0%
PORTO	USA	4.79	1.83	4.24	0.5%	0.3%	0.4%
	UK	13.84	6.36	12.17	0.9%	0.5%	0.7%
	National	13.19	7.58	15.07	0.8%	0.6%	0.7%

It should be noted that a formula corresponding to a simple random sample was used for calculating the error margin, since the weight of the crossing variables was unknown.
For a more detailed reading of the error margins see the report in Annex 4.1.

Weighting

Results were adjusted so that after the adjustment the sample observations weighted by the Week day and Period variable modes had the same weight within each operator, server and municipality.

Technical Architecture, Operating Mode and Component Detail

• Technical Architecture

TEMS™ Automatic was the technical solution used to assess the quality of the mobile broadband access service.

TEMS™ Automatic is an autonomous system that makes it possible to assess the quality perceived by the end user. It provides a large volume of information that can be used for a clear and precise statistical comparison of the performance of several mobile operators or several commercial offerings within a mobile operator. It also makes it possible to obtain details for the identification/resolution of problems and analyses. Measurements are carried out without human intervention.

The architecture of the TEMS™ Automatic system implemented for this study comprised:

a. 30 remote measurement units: MTU

b. 1 central server for data collection and processing

c. 1 FTP/HTTP server installed at FCCN

d. 2 international servers (Houston, London)

Figure 4 - Test Architecture

(Click to enlarge image)

The remote measurement units (from here on designated MTU) were placed motionless in the indoor and outdoor accesses, where they carried out the tests automatically during the previously determined schedules.

Tests were carried out by assessing the connections from the MTU to each of the three servers. After the MTU has collected the test data, this information is sent to the central server, for post-processing.

The work orders were sent to the MTUs automatically, once they were installed in the various accesses. The collected data is also sent to the central server automatically, at regular intervals.

Data post-processing is carried out when this information is received at the central server, where it is added to the system’s database for later analysis.  From this point a massively populated database was created using all collected data that could allow a robust statistical comparison between operators.

• Operating mode

MTU placement was preceded by coverage verification, guaranteeing a minimum RSCP – Received Signal Code Power of -90 dBm for the UMTS signal. The existence of HSDPA technology for the three operators was also confirmed.

The tests were then carried out for one week in each location, fulfilling the same criteria for all accesses that were part of the sample for the cities of Faro, Lisbon and Porto, 60 locations in each municipality in a total of 180.

Units were kept motionless (static) and unobstructed during the collection period.

The test sequences carried out by the measurement units used software applications that made it possible to assess the indicators presented in this study for each operator, using connections to all national and international servers.

The same propagation conditions were guaranteed for the three operators by using a single 3G data card with an external antenna in the measurement unit. Tests were separated by minimum time intervals by using the measurements unit’s ''SIM Multiplexing'' functionality. Thus, a SIM card of each operator used alternately a single 3G data card for transferring the HTTP page, for the upload or download of the binary FTP file, for carrying out the Ping test or the Streaming tests to the targeted test servers.

• Component detail

MTU remote measurement unit

The MTU750 unit was used in the study.

A single external antenna connected to the unit’s data card guarantees exactly the same radio conditions for the 3 operators under analysis. Each unit was geographically referenced in order to localise the indoor or outdoor access.

The data card in the MTU750 is compatible with 3GPP specifications, allowing HSDPA speeds up to 7.2 Mbps and HSUPA up to 2Mbps (Option Globetrotter data card used in MTU750 is a 7.2 Mbps/2Mbps capable data card).

The MTU had 3 SIM cards installed in order to carry out the tests for each operator (Optimus, Vodafone and TMN).

Figure 5 - MTU750

(Click to enlarge image)

National and International FTP/HTTP Servers

In order to assess the indicators for each operator, the MTUs carried out HTTP, FTP, Streaming and Ping tests to a Portuguese server and 2 international servers with the following characteristics:

Portugal (Lisbon FCCN): 

Sun Solaris 10

RAM 16GB

USA (Houston):  

Microsoft Windows Server 2003 SP2

Intel Xeon CPU 2.4GHz

RAM 2GB 

UK (London):   

Microsoft Windows Server 2003 SP2
     
Intel Core2 Duo CPU 2.93GHz
     
RAM 2GB

 
¹ The operators have different understandings of the maximum download speed of the offering under study. Although the commercial offering analysed indicates maximum downlink speeds up to 2 Mbps, and 384 Kbps uplink speeds, the profiles recorded in the HLR have different speeds for each operator. TMN’s SIM cards have defined speeds of 2560 Kbps for downlink and 5824 Kbps for uplink. Vodafone sets the profiles for its SIM cards with maximum speeds of 2432 Kbps for downlink and 384 Kbps for uplink. Optimus has defined the maximum speeds of 2432 Kbps for downlink and 576 Kbps for uplink. It is possible to see that all operators allow higher download speeds than those commercially advertised, and that TMN and Optimus also allow higher uplink speeds than those defined commercially.
² High-Speed Downlink Packet Access
³ Measurements were interrupted between 23 December 2009 and 3 January 2010 since this was considered to be an atypical period.
⁴ The Absolute Precision of results, also named (Absolute) Error Margin, corresponds to half the amplitude of the confidence interval for the variable under study. Relative Precision corresponds to the Absolute Precision divided by the estimate obtained for that variable.