Methodology and Criteria for Evaluating E-Services : The Case of Azerbaijan

The enhancement of evaluation methods and development of effective public administration mechanisms are crucial for providing effective e-government services. This article explores the international practice in the field of e-service evaluation. The organization and provision of e-services in government entities, as well as the evaluation regarding the organization and functioning of information systems are implemented by the “ASAN service” system in the Azerbaijan case. The evaluation of organization and functioning of e-services is conducted by considering three determined criteria: the level of digitalization and relevance, the level of information openness and accessibility, and the level of convenience of use. Evaluation of eservices allows for improving the level of services and the development of feedback mechanisms. Currently, much attention is focused on the development of citizen-centered services.


Introduction
Achievements in the field of transformation of public administration, the building of citizencentered e-government, provision of e-services, as well as the development of effective administrative mechanisms with the application of ICT in most developed and developing countries are all highly appreciated.The modern economy is characterized by two tendencies found in the e-service concept: the shift from goods to services and the rapid development of information economy and electronic systems (Ostašius & Petravičiūtė, 2010).By e-services, it is assumed to be the organization of services in various fields in electronic format organized with the application of ICT, without loss of time and transparency.
The growing demand of users for these services boosts the development of e-services in order to respond more effectively to their needs.E-services must be simple and easily accessible for a wide audience of users.In this regard, the evaluation of electronic services is one of the most important issues.Through experience, usually, common characteristic problems are found to pertain to such systems, and from this point of view, the development of projects specific to the field of e-services are, therefore, of no importance.It can be said that, it is more reasonable to agree and to cooperate in order to find common solutions to be applied by all interested parties.In this regard, the cooperation and the application of a common e-services model allows for finding more efficient solutions by considering the competencies of all parties providing services (Lindgren & Jansson 2013).
In recent periods of rapid development of ICT, purposeful works are being performed towards e-government building and the organization of various e-services in different corners of the world.An e-government project has already been adopted in Azerbaijan, a national strategy has been developed and carried out within the framework of the "Electronic Azerbaijan" Programme since 2003 (E-Azerbaijan, 2003(E-Azerbaijan, -2012)).The main purpose of the project implementation is the provision of services quality, timeliness, transparency, efficiency, and accessibility for citizens with the "one window" principle in government and other government institutions with the application of continuously developing ICT.
Nowadays, e-services are developed by applying new technologies, innovations, standards, and learning from established best practices to increase the efficiency of e-government.The rapid growth of the number of new e-services promotes this evaluation and their multilevel measurements.E-service evaluation models make it possible to measure e-services and their systems in different ways and dimensions.This article explores an international practice regarding the evaluation of e-services and the experience of Azerbaijan.

International Practice on E-Service Evaluation
The achievements in the implementation of initiatives related to e-government have turned egovernment into a worldwide phenomenon.Governments adopt various strategies and state programs in order to implement e-government development policy.Notwithstanding all these efforts, the development of ICT and new services, the requirements posed to the provision of citizen-oriented services, etc. are accompanied with challenges and problems at local and regional levels.For example, the requirements posed to e-services that alter with the lapse of time necessitate the development of e-government strategies and models.
The development of e-government models can be divided into three stages: starting from Gov. 1.0 model in the world in 1995, the application of Gov. 2.0 model in 2005-2010, and the application of Gov. 3.0 model since 2015 (E-Gov. concepts, 2008;Meijer, et al. 2012;Government 3.0, 2016).
At present, the Gov. 2.0 platform is being utilized in order to provide more citizen-oriented e-government services.Web 2.0 technologies are used in order to implement and provide government-and citizen-oriented services.It should be noted that, the Gov 3.0 model that is proposed as a concept is an incentive of customizing the services in accordance with desires and needs of each citizen; in other words, it is a web-based semantic government incentive providing smart services.E-government models differ from each other according to various characteristics of their stages of development (E-Gov. concepts, 2008;Government 3.0, 2016).
As one of the leading countries according to the government development index developed by the United Nations (UN), South Korea has adopted an e-government 2020 strategy and aims to provide citizen-centered e-government services with the application of advanced technologies (E-Government Survey 2014Survey , 2016;;Government 3.0, 2016).According to the experience of South Korea, the Gov. 3.0 concept is a citizen-centered government service and its aim is to make human lives more comfortable, to create new job places with open government data, and to promote cooperation and communication among government entities.By making government more service-oriented, competent and transparent, Gov. 3.0 is oriented towards the improvement of the satisfaction level of citizens (Government 3.0, 2016).Ray & Rao (2005) have proposed a method for the accurate evaluation of efficiency level of services provided as a result of the implementation of the e-government project.It is proposed to employ the analytical hierarchy approach as a tool for evaluating e-services provided.Ostašius & Petravičiūtė (2010) attribute much attention mainly to the modelling of egovernment services in their study.This approach is based on a reference model of serviceoriented architecture.The model is oriented towards the stimulation of cooperation among entities participating in the process of the provision of e-services and the improvement of the efficiency in their functioning thanks to the development of joint services.The proposed model has been tested in e-services of transport vehicles registration and can be used for developing other services in egovernment.The proposed e-service model allows evaluation of the maturity and completeness of an e-service system, as well as the ability to compare it with other similar e-services (Ostašius & Petravičiūtė, 2010).
In their study, Lindgren & Jansson (2013) discuss the definitions used for the study of e-services issues related to the employment and development of e-government services.The conceptual foundations of e-services are explored and e-government services are studied in three aspects: service, electronic, and public (Lindgren & Jansson, 2013).Zaidi, et al. (2013) explore various aspects of the provision of e-government services, and an approach is proposed for evaluating the trustworthiness and the efficiency of e-services provided from the point of view of citizens.Existing models for efficiency evaluation such as SERVQAUAL, E-S-Qual, D&M, etc. have been systematically analyzed, and a new conceptual model has been proposed (Zaidi, et al. 2013).
Butt (2014) reviews the evaluation of result-oriented e-government by considering citizen interests.The study presents the evaluation of e-government websites as representing the evaluation of e-government.The literature review states that, these kind of studies are usually conducted in order to evaluate the functioning of government institutions.The evaluation of egovernment is implemented in three stages in this research.The first stage discusses the indicators for e-government evaluation.The second stage considers the collection of citizen opinions on each indicator for conducting an experiment.The approach is experimentally tested in a selected government entity at the third stage.As this approach is outcome-oriented, the reports prepared based on results are sent to interested parties.The results of the study have shown that the evaluation conducted in a government institution allows analyzing and critically evaluating the process at a particular level in the government institution (Butt, 2014).Hasan (2015) has made a literature analysis of studies conducted in the e-government sphere.The analysis of the studies are conducted with the intent of research improvement by taking into account the regional context, the perspective of studies, the method of data analysis, and the analysis of research approaches.The outcome of this study is important in terms of exploring egovernment services and determining future research directions.The results obtained in the study cover the research works related to e-government services, service development, evaluation, and mutual communication.The research works considered the topic from the point of view of advanced and developing countries, and empirical data analysis methods were used (Hasan, 2015).Taherdoost, et al. (2015) summarized e-services and explored the conceptual perspectives comprehensively.The scientific-theoretical foundations of e-services are studied in terms of conducting future empirical studies and e-service users are divided into categories, as well.Asad, Khazaei, et al. (2016) investigate the conceptual foundations of the importance of integration of provided services in the implementation of e-government, and a literature analysis is also completed.The integration of e-services, as well as the definitions, problems, analytical and conceptual approaches are explored.As a result, a model is proposed that enables the specification of the importance of services integration in the implementation of e-government (Asad, & Khazaei, et al. 2016).Malik, et al. (2016) consider the evaluation of citizen satisfaction during the use of e-government services.Citizen satisfaction plays a crucial and decisive role for systematic use of e-government services and influences the failure or success of e-government projects.The results obtained in this research evaluated the level of citizen satisfaction in Pakistan (Malik, et al. 2016).
With the passage of time, the development of e-government models is carried out based on proposals and social innovations for the purpose of e-services to be evaluated by citizens and governments regularly in order to improve the level of services provided.This process results in the generation of new e-government models.
Considering the European experience, the evaluation of e-services in the UN model is executed based on a 4-stage (emerging, enhanced, transactional, connected) development model according to the online service index (E-Government Survey 2014Survey , 2016)).It is to be mentioned that, although e-services are being evaluated based on various criteria in international practice and various local and regional approaches are present, a unified evaluation criterion does not exist.

Criteria for E-Services Evaluation in Azerbaijan
Purposeful works are being executed aiming for the development of e-services in Azerbaijan.It is known that particular requirements are posed to the implementation of each e-service.These requirements can be classified as follows: 1. Facilitation of the use of e-services -the provision of the name of a provided service by relevance, accurate list of documents required for providing the service, the electronic form of corresponding documents (application, forms, and others) in the e-service section of the official internet portal, and free use of this section by people without obstacles; 2. Facilitation of document submission -the provision of the acceptance of scanned copies of applications and required documents in electronic format, the notification of acceptance to a person sending the documents, and the review of those documents in accordance with the procedure and time specified by the legislation; 3. User informing -the facilitation of obtaining information regarding the procedure of eservice provision for each user and the presentation of results; 4. Integration facilitation -building communication for mutual information exchange with other government entities for the purpose of e-services provision; 5. Security provision -adoption of necessary technical security measures in order to provide the security of operations conducted via the e-service section and personal information; 6. Presence of administrative regulation -information resources must be developed by a government institution that will provide the service, and also the performance and actions of all institutions participating in the service delivery must be described accurately and be open to the public; 7. Payments for e-service -the provision of payment opportunity in electronic format in real time; 8. Use of e-signature -authentication of e-service users, the submission of applications and responses, and signing the documents sent in electronic format to central executive authorities.
The organization and provision of e-services in government entities, the procedure for evaluation of the organization, and the functioning of information systems is carried out on an annual basis based on the principles of lawfulness, objectiveness, transparency, and professionalism by the Azerbaijan Service and Assessment Network shortly called "ASAN service" (State Agency for Public Services; ASAN service).The main purpose of the evaluation is as follows: 1. Organization and provision of e-services, as well as the study of the current situation and conducted works regarding the organization and functioning of information systems; 2. Determination of information sources necessary for organizing and providing e-services; 3. Supervision on the implementation of information systems to be performed in accordance with approved projects, technical norms, standards, and requirements; 4. Provision of mutual integration of information systems of government entities; 5. Supervision of the acceleration of the connection process to developed infrastructure for the purpose of information exchange between government entities and information systems; 6.The investigation of opportunities of simplifying the procedure of provision of state services (with regard to required documents and periods); 7. Determination of information systems necessary for the implementation of state services; 8. Acceleration of the process of e-services organization; 9. Organization and provision of e-services, methodical and operational assistance to government entities on the organization and functioning of information systems; 10.Provision of transparency in the operation of government entities; 11.Exploration of opportunities of the provision of modern and flexible administration in this field (State Agency for Public Service).
The evaluation of the level of organization and provision of e-services is carried out by a State Agency according to three criteria and sub-criteria in the following directions (State Agency for Public Service):  Level of digitalization and relevance;  Level of information openness and accessibility;  Level of convenience of use.
The level of digitalization and connectedness consists of five sub-criteria and it is evaluated with 0, 1, and 2 points in terms of digitalization, integration of electronic information resources between other government entities, the use of electronic information systems, and other issues.The subcriteria are as follows: 1. Using electronic information systems during the organization and provision of e-services (acceptance of application in regard to e-services, processing in government entities, and delivery of results); 2. Using internal electronic information resources (databases, information-search systems, registries, and other information resources) during the provision of e-services; 3. Using electronic information resources (databases, information-search systems, registries, and other information resources) of other government institutions regarding the provision of e-services; 4. Provision of the use of e-signature while providing e-services; 5. Presentation of the results of e-service provision.
The level of information openness and accessibility consists of the following four sub-criteria: 1. Presence of administrative regulation of e-service; 2. Provision of service in the section of state services in official internet portal of the government entity providing the particular service; 3. Presentation of comprehensive, up-to-date, and accurate information regarding a service in "E-registry of State Services"; 4. Availability of opportunities to obtain comprehensive, up-to-date, and accurate information on the official internet page of a government entity regarding the rules of the e-services usage.
The level of convenience of use criterion consists of the following seven sub-criteria and evaluation is done in accordance with the corresponding sub-criteria: 1.The opportunity of acceptance of information required for the use of service in electronic format; 2. Availability of sample documents (application, forms, and others) in electronic format in the section of state services of the portal and free use of this section by people without any obstacles; 3. Delivery of notification of acceptance to a person who submitted the documents in electronic format (who sent the request for informative services); 4. Facilitation of opportunities to obtain electronic information regarding the procedure of e-service provision for users; 5. Facilitation of payments in electronic format in real time for the provision of e-service; 6. Review of the documents submitted in electronic format in accordance with the procedure and periods as expected by the legislation; 7. Provision of the review of complaints and suggestions regarding services received via "ASAN appeal" system (ASAN service).
It is to be noted that, the evaluation of e-services is administered in three stages: submission of requests, implementation of spot review of corresponding government institutions providing e-services, and the test of services via information systems (test use of service).
Statistical indicators regarding the organization and provision of services by "ASAN service" in 2013-2016 is shown in Figure 1: According to specified sub-criteria and the accompanying stages, electronic service was assessed with 0, 1, and 2 points by experts and the total points on percentages are correspondingly evaluated as follows:  50% and lower -non-satisfactory;  51% to 70% -satisfactory;  71% to 90% -good;  91% to 100% -excellent.E-services provided by corresponding entities of executive authorities have been evaluated according to relevant indicators and 79 services have been added to the list of excellent services.
It must be noted that the evaluation is carried out in accordance with the rules prepared by the State Agency for Public Service.Various institutions, community representatives, and experts have been engaged in the preparation of rules.These rules, which are annually updated, have been prepared in line with recommendations from the European Union, UN, and international practice.
As a result of the evaluation of e-services provided by the State Agency of Public services, the level of digitalization of services, integration, relevance, and convenience of use have been improved.Also, the user feedback has been organized in a more comprehensive way and the best practices of various institutions have been shared with other institutions.Moreover, specific proposals are submitted to government institutions oriented towards the improvement of the level of e-services, which are provided by those institutions (State Agency for Public Service).
According to the results of the report by the UN (E-Government Survey 2016) on the development of e-government implementation for the purpose of measuring the efficiency of egovernment in delivering main economic and social services to people, Azerbaijan has advanced 12 steps in the e-government development index and so it occupies 56 th place among 193 countries in 2016 in comparison with 2014.According to the regional ranking, Azerbaijan occupies 11 th place in the Asian region, and 4th place among CIS countries after Kazakhstan, Russia, and Belarus.Azerbaijan has gained 0.6780 points for e-participation index in 2016 and occupies 47 th place among 193 countries (E-Government Survey 2016).These results reflect that the performance of the State Agency for Public Service is highly appreciated, and at present, the international experience in the sphere of providing citizen-oriented services are closely studied for the purpose of improving e-services.
Around 6 million people have received ''ASAN service'' since its inception during the first three years.Mobile ''ASAN Service'' is a top innovative approach developed by the State Agency for ensuring the universal accessibility of services for the citizens throughout the country.This service ensures a proactive approach by bringing state services to the people.It involves well-equipped large buses that travel to the regions to deliver services in rural, remote, and hard-to-reach areas that do not host ''ASAN Service'' centers.
The centers of ''ASAN Service'' aim to enhance transparency, to strengthen the fight against corruption, to reduce citizens' additional costs and time loss, and to provide easy access to public services.The transition to an information society is not confined with the reforms applied in public administration.It requires the development of democratic principles, e-democracy mechanisms, and also reconstruction based on the interests of citizens.
The achievements in the implementation of initiatives related to e-government have turned e-government into a world phenomenon.Governments adopt various strategies and state programs in order to implement e-government development policy.Despite all of these efforts, the development of ICT, and new services, the requirements posed to the provision of citizen-oriented services, etc. are accompanied with challenges and problems at local and regional levels.For example, the requirements posed to e-services that alter time intervals necessitate the development of e-government strategies and models.

Conclusion
At present, ICT facilities are broadly used while providing state services and implementing public functions.In this regard, if it is possible to say so, the application of ICT takes the lead of the evaluation of e-service systems.The evaluation of e-services is practically related to the application and use of the system.There exists a need for a systematic approach for analyzing, monitoring, and measuring the effectiveness of e-government services.The systematic approach plays an important role in the evaluation of the efficiency of e-services.At the present time, much attention is attributed to the study of citizen-centered services.This article explores existing international practice in the field of e-services evaluation.The experience of Azerbaijan in the sphere of the provision of e-services is investigated.The article explores the criteria for measuring e-services provided by "ASAN service", as well as proposed approaches to the evaluation of the level of organization and provision of e-services in government institutions.Following the evaluation of e-services, the level of digitalization of services, the level of integration and relevance, and the convenience of use of services, specific proposals are received regarding the level of e-services and their improvement.
The importance of citizen opinion in the process of the evaluation of e-government services cannot be denied, as citizens are the main interested party while using e-services.In this regard, the development of feedback mechanisms between citizens and state institutions bears great importance, and the use of feedback indicators during the evaluation of the efficiency of e-services will facilitate more effective decision-making and the development of effective solutions.