国产无码综合区,色欲AV无码国产永久播放,无码天堂亚洲国产AV,国产日韩欧美女同一区二区

<rp id="agg5y"></rp><sub id="agg5y"></sub>

<style id="agg5y"></style>

【Hadoop綜合實(shí)踐】手機(jī)賣場(chǎng)大數(shù)據(jù)綜合項(xiàng)目分析

2年前作者：大數(shù)據(jù)小禪分類：Toy博客閱讀(24)違法舉報(bào)

這篇具有很好參考價(jià)值的文章主要介紹了【Hadoop綜合實(shí)踐】手機(jī)賣場(chǎng)大數(shù)據(jù)綜合項(xiàng)目分析。希望對(duì)大家有所幫助。如果存在錯(cuò)誤或未考慮完全的地方，請(qǐng)大家不吝賜教，您也可以點(diǎn)擊"舉報(bào)違法"按鈕提交疑問(wèn)。

?? 本文章實(shí)現(xiàn)了基于MapReduce的手機(jī)瀏覽日志分析
?? 文章簡(jiǎn)介：主要包含了數(shù)據(jù)生成部分，數(shù)據(jù)處理部分，數(shù)據(jù)存儲(chǔ)部分與數(shù)據(jù)可視化部分
?? 【本文僅供參考??！非唯一答案】其中需求實(shí)現(xiàn)的方式有多種，提供的代碼并非唯一寫(xiě)法，選擇適合的方式即可。

手機(jī)日志分析需求

本文主要實(shí)現(xiàn)以下需求

編寫(xiě)數(shù)據(jù)生成器生成1G~10G大小的數(shù)據(jù)，字段必須包括id,日期,手機(jī)號(hào)碼、型號(hào)、操作系統(tǒng)字段。
需要將手機(jī)號(hào)碼4~9為掩碼處理。
分析2021年、2022年操作系統(tǒng)市場(chǎng)占比、手機(jī)型號(hào)市場(chǎng)占比情況
分析2022年手機(jī)運(yùn)營(yíng)商市場(chǎng)占比情況
分析數(shù)據(jù)存儲(chǔ)到HDFS集群/ana/phone節(jié)點(diǎn)下面
將分析結(jié)果存儲(chǔ)到Mysql,并進(jìn)行數(shù)據(jù)可視化

數(shù)據(jù)生成工具類

手機(jī)號(hào)碼隨機(jī)生成
可以采用隨機(jī)數(shù)生成的方式，結(jié)合三大運(yùn)營(yíng)商的號(hào)碼前三位數(shù)為規(guī)則進(jìn)行生成代碼如下

/**
 * @Description  生成三大運(yùn)營(yíng)商的手機(jī)號(hào)
 */
/**
 * 中國(guó)移動(dòng)手機(jī)號(hào)段：
 * 134、135、136、137、138、139、147、150、151、152、157、158、159、172、178、182、183、184、187、188、198、1703、1705、1706
 * 中國(guó)聯(lián)通手機(jī)號(hào)段：
 * 130、131、132、145、155、156、166、171、175、176、185、186、1704、1707、1708、1709
 * 中國(guó)電信手機(jī)號(hào)段：
 * 133、153、173、177、180、181、189、191、193、199、1700、1701、1702
 * 騰訊云API https://market.cloud.tencent.com/products/31101
 */
public class PhoneNumberGenerator {
    //生成一萬(wàn)個(gè)手機(jī)號(hào)碼，只需將 generatePhoneNumbers 方法中的參數(shù) count 修改為 10000 即可
    //移動(dòng)
    private static final String[] CHINA_MOBILE_PREFIX = {"134", "139", "150", "151", "182"};
    //聯(lián)通
    private static final String[] CHINA_UNICOM_PREFIX = {"130","155","186"};
    //電信
    private static final String[] CHINA_TELECOM_PREFIX = {"133","153","180","181","189"};

    public static void main(String[] args) {
        String phoneNumbers = generatePhoneNumbers(1);
            System.out.println(phoneNumbers);
    }

    public static String generatePhoneNumbers(int count) {
        String phoneNumber=null;
        Random random = new Random();
        for (int i = 0; i < count; i++) {
            String prefix;
            int operatorIndex = random.nextInt(3);
            switch (operatorIndex) {
                case 0:
                    prefix = CHINA_MOBILE_PREFIX[random.nextInt(CHINA_MOBILE_PREFIX.length)];
                    break;
                case 1:
                    prefix = CHINA_UNICOM_PREFIX[random.nextInt(CHINA_UNICOM_PREFIX.length)];
                    break;
                default:
                    prefix = CHINA_TELECOM_PREFIX[random.nextInt(CHINA_TELECOM_PREFIX.length)];
            }
            phoneNumber = prefix + generateRandomNumber(random, 11 - prefix.length());
        }
        return replaceCharacters(phoneNumber,3,8);
    }
    private static String replaceCharacters(String input, int startIndex, int endIndex) {
        if (input == null || input.length() < endIndex) {
            return input;
        }

        StringBuilder sb = new StringBuilder(input);

        for (int i = startIndex; i <= endIndex; i++) {
            sb.setCharAt(i, '*');
        }

        return sb.toString();
    }
    private static String generateRandomNumber(Random random, int length) {
        StringBuilder sb = new StringBuilder();
        for (int i = 0; i < length; i++) {
            sb.append(random.nextInt(10));
        }
        return sb.toString();
    }
}

運(yùn)營(yíng)商解析的其中一種方式【采用接口分析】
這里可以使用鵝廠或者其他廠商開(kāi)發(fā)的接口進(jìn)行運(yùn)營(yíng)商識(shí)別申請(qǐng)獲取對(duì)應(yīng)的秘鑰即可例子如下


public class PhoneOperator {
    public static String calcAuthorization(String source, String secretId, String secretKey, String datetime)
            throws NoSuchAlgorithmException, UnsupportedEncodingException, InvalidKeyException {
        String signStr = "x-date: " + datetime + "\n" + "x-source: " + source;
        Mac mac = Mac.getInstance("HmacSHA1");
        Key sKey = new SecretKeySpec(secretKey.getBytes("UTF-8"), mac.getAlgorithm());
        mac.init(sKey);
        byte[] hash = mac.doFinal(signStr.getBytes("UTF-8"));
        String sig = new BASE64Encoder().encode(hash);

        String auth = "hmac id=\"" + secretId + "\", algorithm=\"hmac-sha1\", headers=\"x-date x-source\", signature=\"" + sig + "\"";
        return auth;
    }

    public static String urlencode(Map<?, ?> map) throws UnsupportedEncodingException {
        StringBuilder sb = new StringBuilder();
        for (Map.Entry<?, ?> entry : map.entrySet()) {
            if (sb.length() > 0) {
                sb.append("&");
            }
            sb.append(String.format("%s=%s",
                    URLEncoder.encode(entry.getKey().toString(), "UTF-8"),
                    URLEncoder.encode(entry.getValue().toString(), "UTF-8")
            ));
        }
        return sb.toString();
    }

    public static void main(String[] args) throws NoSuchAlgorithmException, UnsupportedEncodingException, InvalidKeyException {
        //云市場(chǎng)分配的密鑰Id
        String secretId = "xx";
        //云市場(chǎng)分配的密鑰Key
        String secretKey = "xx;
        String source = "market";

        Calendar cd = Calendar.getInstance();
        SimpleDateFormat sdf = new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss 'GMT'", Locale.US);
        sdf.setTimeZone(TimeZone.getTimeZone("GMT"));
        String datetime = sdf.format(cd.getTime());
        // 簽名
        String auth = calcAuthorization(source, secretId, secretKey, datetime);
        // 請(qǐng)求方法
        String method = "POST";
        // 請(qǐng)求頭
        Map<String, String> headers = new HashMap<String, String>();
        headers.put("X-Source", source);
        headers.put("X-Date", datetime);
        headers.put("Authorization", auth);

        // 查詢參數(shù)
        Map<String, String> queryParams = new HashMap<String, String>();
        queryParams.put("mobile","XXX");
        // body參數(shù)
        Map<String, String> bodyParams = new HashMap<String, String>();
        // url參數(shù)拼接
        String url = "https://service-8c43o60c-1253285064.gz.apigw.tencentcs.com/release/sms";
        if (!queryParams.isEmpty()) {
            url += "?" + urlencode(queryParams);
        }

        BufferedReader in = null;
        try {
            URL realUrl = new URL(url);
            HttpURLConnection conn = (HttpURLConnection) realUrl.openConnection();
            conn.setConnectTimeout(5000);
            conn.setReadTimeout(5000);
            conn.setRequestMethod(method);

            // request headers
            for (Map.Entry<String, String> entry : headers.entrySet()) {
                conn.setRequestProperty(entry.getKey(), entry.getValue());
            }

            // request body
            Map<String, Boolean> methods = new HashMap<>();
            methods.put("POST", true);
            methods.put("PUT", true);
            methods.put("PATCH", true);
            Boolean hasBody = methods.get(method);
            if (hasBody != null) {
                conn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");

                conn.setDoOutput(true);
                DataOutputStream out = new DataOutputStream(conn.getOutputStream());
                out.writeBytes(urlencode(bodyParams));
                out.flush();
                out.close();
            }

            // 定義 BufferedReader輸入流來(lái)讀取URL的響應(yīng)
            in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
            String line;
            String result = "";
            while ((line = in.readLine()) != null) {
                result += line;
            }
            System.out.println(result);
        } catch (Exception e) {
            System.out.println(e);
            e.printStackTrace();
        } finally {
            try {
                if (in != null) {
                    in.close();
                }
            } catch (Exception e2) {
                e2.printStackTrace();
            }
        }
    }
}

結(jié)果如下（另一種方式為：直接根據(jù)前三位手機(jī)號(hào)進(jìn)行判斷）
【Hadoop綜合實(shí)踐】手機(jī)賣場(chǎng)大數(shù)據(jù)綜合項(xiàng)目分析

模擬數(shù)據(jù)生成類

數(shù)據(jù)生成器 id,日期,手機(jī)號(hào)碼、型號(hào)、操作系統(tǒng)


/**
 * @Description
 *  數(shù)據(jù)生成器  id,日期,手機(jī)號(hào)碼、型號(hào)、操作系統(tǒng)
 *  id:UUID 隨機(jī)生成  日期:2021 2022 手機(jī)號(hào)碼:三大運(yùn)營(yíng)商  型號(hào):Apple HuaWei Oppo Vivo Meizu Nokia  操作系統(tǒng):Apple ios Harmony Samsung
 *  1.分析2021年、2022年操作系統(tǒng)市場(chǎng)占比、手機(jī)型號(hào)市場(chǎng)占比情況
 *  2.分析2022年手機(jī)運(yùn)營(yíng)商市場(chǎng)占比情況
 *  3.分析數(shù)據(jù)存儲(chǔ)到HDFS集群/ana/phone節(jié)點(diǎn)下面
 *  4.將分析結(jié)果存儲(chǔ)到Mysql,并進(jìn)行數(shù)據(jù)可視化
 */
public class DataGenerator {

    public static void main(String[] args) {
        try {
            BufferedWriter writer = new BufferedWriter(new FileWriter("data/phone.log"));
            for (int i = 0; i < 1000; i++) {
                //UUID隨機(jī)生成   id,日期,手機(jī)號(hào)碼、型號(hào)、操作系統(tǒng)
                String id = UUID.randomUUID().toString();
                String date = getRandomDate();
                String phoneNumber = PhoneNumberGenerator.generatePhoneNumbers(1);
                String model = getRandomModel();
                String operatingSystem = getRandomOperatingSystem();
                String line = id + "," + date + "," + phoneNumber + "," + model + "," + operatingSystem;
                writer.write(line);
                writer.newLine();
            }
            writer.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    private static String getRandomDate() {
        Random random = new Random();
        int year = random.nextInt(2) == 0 ? 2021 : 2022;
        int month = random.nextInt(12) + 1;
        int dayOfMonth;

        if (month == 2) {
            dayOfMonth = random.nextInt(28) + 1;
        } else if (month == 4 || month == 6 || month == 9 || month == 11) {
            dayOfMonth= random.nextInt(30) + 1;
        } else {
            dayOfMonth= random.nextInt(31) + 1;
        }

        return year + "-" +
                (month < 10 ? "0" : "") +
                month+ "-" +
                (dayOfMonth<10? "0":"")+
                dayOfMonth ;
    }

    private static String getRandomPhoneNumber() {
        Random random = new Random();
        StringBuilder phoneNumber = new StringBuilder("1");

        for (int i = 0; i < 10; i++) {
            phoneNumber.append(random.nextInt(10));
        }

        return phoneNumber.toString();
    }

    private static String getRandomModel() {
        String[] models = {"Apple", "HuaWei", "Oppo", "Vivo", "Meizu", "Nokia"};
        return models[new Random().nextInt(models.length)];
    }

    private static String getRandomOperatingSystem() {
        String[] operatingSystems = {"Apple", "HarmonyOS", "Samsung","iOS"};
        return operatingSystems[new Random().nextInt(operatingSystems.length)];
    }
}

結(jié)果如下
【Hadoop綜合實(shí)踐】手機(jī)賣場(chǎng)大數(shù)據(jù)綜合項(xiàng)目分析

MapReduce程序需求編寫(xiě)

分析2021年、2022年操作系統(tǒng)市場(chǎng)占比、手機(jī)型號(hào)市場(chǎng)占比情況


/**
 * @Description
 */
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Partitioner;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class PhoneOSAnalysis {
    private static int totalCount;
    private static int lineCount2021 = 0;
    private static int lineCount2022 = 0;
    public static class TokenizerMapper extends Mapper<Object, Text, Text, DoubleWritable> {
        private final static DoubleWritable one = new DoubleWritable(1);
        private Text word = new Text();

        public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
            String[] fields = value.toString().split(",");
            if (fields.length >= 5) {
                // 操作系統(tǒng)市場(chǎng)占比
                word.set(fields[1].substring(0, 4) + "-OS-" + fields[4]);
                context.write(word, one);
                // 手機(jī)型號(hào)市場(chǎng)占比
                word.set(fields[1].substring(0, 4) + "-Model-" + fields[3]);
                context.write(word, one);
            }
        }
    }

    public static class MarketShareReducer extends Reducer<Text, DoubleWritable, Text, DoubleWritable> {
        private DoubleWritable result = new DoubleWritable();

        public void reduce(Text key, Iterable<DoubleWritable> values, Context context)
                throws IOException, InterruptedException {
            double sum = 0;
            for (DoubleWritable val : values) {
                //這里會(huì)根據(jù)分組的key來(lái)計(jì)算sum
                sum += val.get();
            }
            int yearTotalCount = key.toString().contains("2021") ? lineCount2021 : lineCount2022;
            double percentage = sum / yearTotalCount;
            result.set(percentage);
            context.write(key, result);
        }
    }

    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        FileSystem fs = FileSystem.get(conf);
        Path inputPath = new Path("data/phone.log");
        FSDataInputStream inputStream = fs.open(inputPath);
        try (BufferedReader reader = new BufferedReader(new InputStreamReader(fs.open(inputPath)))) {
            String line;
            while ((line = reader.readLine()) != null) {
                if (line.contains("2021")) {
                    lineCount2021++;
                } else if (line.contains("2022")) {
                    lineCount2022++;
                }
            }
        }

//        totalCount = Math.max(lineCount2021, lineCount2022);

        Job job = Job.getInstance(conf, "market share analysis");
        job.setJarByClass(PhoneOSAnalysis.class);
        job.setMapperClass(TokenizerMapper.class);

        // 設(shè)置自定義分區(qū)器
        job.setPartitionerClass(CustomPartitioner.class);
        job.setNumReduceTasks(2);

        job.setReducerClass(MarketShareReducer.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(DoubleWritable.class);
        Path outputPath = new Path("data/result");
        if (fs.exists(outputPath)) {
            fs.delete(outputPath, true);
        }
        FileInputFormat.addInputPath(job, new Path("data/phone.log"));
        FileOutputFormat.setOutputPath(job, new Path(String.valueOf(outputPath)));
        //        TextInputFormat.addInputPath(job, new Path("hdfs://192.168.192.100:8020/"));
//        TextInputFormat.outInputPath(job, new Path("hdfs://192.168.192.100:8020/"));
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }

    public static class CustomPartitioner extends Partitioner<Text, DoubleWritable> {

        @Override
        public int getPartition(Text key, DoubleWritable value, int numPartitions) {
            // 根據(jù)年份進(jìn)行分區(qū)
            if (key.toString().contains("2021")) {
                return 0;
            } else {
                return 1;
            }
        }
    }
}

分析2022年手機(jī)運(yùn)營(yíng)商市場(chǎng)占比情況


/**
 * @Description
 */
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class OperatorMR {
    private static int lineCount2022 = 0;

    public static class TokenizerMapper extends Mapper<Object, Text, Text, DoubleWritable> {
        private final static DoubleWritable one = new DoubleWritable(1);
        private Text word = new Text();

        public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
            String[] fields = value.toString().split(",");
            if (fields.length >= 3 && fields[1].contains("2022")) {
                // 手機(jī)運(yùn)營(yíng)商市場(chǎng)占比
                word.set(fields[1].substring(0, 4) + "-Operator-" + getCarrier(fields[2]));
                context.write(word, one);
            }
        }

        private String getCarrier(String phoneNumber) {
            String prefix = phoneNumber.substring(0, 3);
            switch (prefix) {
                //"133","153","180","181","189"
                case "133":
                case "153":
                case "180":
                case "181":
                case "189":
                    return "電信";
                //"130","155","186"
                case "130":
                case "155":
                case "186":
                    return "聯(lián)通";
                default:
                    return "移動(dòng)";
            }
        }
    }

    public static class MarketShareReducer extends Reducer<Text, DoubleWritable, Text, DoubleWritable> {
        private DoubleWritable result = new DoubleWritable();

        public void reduce(Text key, Iterable<DoubleWritable> values, Context context)
                throws IOException, InterruptedException {
            double sum = 0;
            for (DoubleWritable val : values) {
                sum += val.get();
            }
            double percentage = sum / lineCount2022;
            result.set(percentage);
            context.write(key, result);
        }
    }

    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        FileSystem fs = FileSystem.get(conf);
        Path inputPath = new Path("data/phone.log");

        try (BufferedReader reader = new BufferedReader(new InputStreamReader(fs.open(inputPath)))) {
            String line;
            while ((line = reader.readLine()) != null) {
                if (line.contains("2022")) {
                    lineCount2022++;
                }
            }
        }

        Job job = Job.getInstance(conf, "PhoneOperator");
        job.setJarByClass(OperatorMR.class);
        job.setMapperClass(TokenizerMapper.class);
        job.setReducerClass(MarketShareReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(DoubleWritable.class);

        Path outputPath = new Path("data/result-phone");

        if (fs.exists(outputPath)) {
            fs.delete(outputPath, true);
        }
//        TextInputFormat.addInputPath(job, new Path("hdfs://192.168.192.100:8020/"));
//        TextInputFormat.outInputPath(job, new Path("hdfs://192.168.192.100:8020/"));
        FileInputFormat.addInputPath(job, new Path("data/phone.log"));
        FileOutputFormat.setOutputPath(job, outputPath);
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}

結(jié)果如下
【Hadoop綜合實(shí)踐】手機(jī)賣場(chǎng)大數(shù)據(jù)綜合項(xiàng)目分析

-將分析結(jié)果存儲(chǔ)到Mysql,并進(jìn)行數(shù)據(jù)可視化

package com.yopai.mrmysql;

/**
 * @Description
 */

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.DoubleWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.db.DBConfiguration;
import org.apache.hadoop.mapreduce.lib.db.DBOutputFormat;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;


public class OPMysqlMR {
    private static int lineCount2022 = 0;

    public static class TokenizerMapper extends Mapper<Object, Text, Text, DoubleWritable> {
        private final static DoubleWritable one = new DoubleWritable(1);
        private Text word = new Text();

        public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
            String[] fields = value.toString().split(",");
            if (fields.length >= 3 && fields[1].contains("2022")) {
                // 手機(jī)運(yùn)營(yíng)商市場(chǎng)占比
                word.set(fields[1].substring(0, 4) + "-Operator-" + getCarrier(fields[2]));
                context.write(word, one);
            }
        }

        private String getCarrier(String phoneNumber) {
            String prefix = phoneNumber.substring(0, 3);
            switch (prefix) {
                case "133":
                case "153":
                case "180":
                case "181":
                case "189":
                    return "電信";
                case "130":
                case "155":
                case "186":
                    return "聯(lián)通";
                default:
                    return "移動(dòng)";
            }
        }
    }

    public static class MarketShareReducer extends Reducer<Text, DoubleWritable, DBOutputWritable, NullWritable> {
        private DoubleWritable result = new DoubleWritable();

        public void reduce(Text key, Iterable<DoubleWritable> values, Context context)
                throws IOException, InterruptedException {
            double sum = 0;
            for (DoubleWritable val : values) {
                sum += val.get();
            }
            double percentage = sum / lineCount2022;
            result.set(percentage);
            context.write(new DBOutputWritable(key.toString(), result.get()), NullWritable.get());
        }
    }

    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();

        // 設(shè)置數(shù)據(jù)庫(kù)連接信息
        String dbUrl = "jdbc:mysql://localhost:3306/blog";
        String dbUsername = "root";
        String dbPassword = "Admin2022!";

        DBConfiguration.configureDB(conf, "com.mysql.jdbc.Driver", dbUrl, dbUsername, dbPassword);

        try (Connection connection = DriverManager.getConnection(dbUrl, dbUsername, dbPassword)) {
            String createTableSql = "CREATE TABLE IF NOT EXISTS operator_market_share(operator VARCHAR(255), market_share DOUBLE)";
            PreparedStatement preparedStatement = connection.prepareStatement(createTableSql);
            preparedStatement.executeUpdate();
        }

        FileSystem fs = FileSystem.get(conf);
        Path inputPath = new Path("data/phone.log");

        try (BufferedReader reader = new BufferedReader(new InputStreamReader(fs.open(inputPath)))) {
            String line;
            while ((line = reader.readLine()) != null) {
                if (line.contains("2022")) {
                    lineCount2022++;
                }
            }
        }

        Job job = Job.getInstance(conf, "PhoneOperator");
        job.setJarByClass(OPMysqlMR.class);

        job.setMapperClass(TokenizerMapper.class);
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(DoubleWritable.class);

        job.setReducerClass(MarketShareReducer.class);
        job.setOutputKeyClass(DBOutputWritable.class);
        job.setOutputValueClass(NullWritable.class);

        // 設(shè)置數(shù)據(jù)庫(kù)輸出
        DBOutputFormat.setOutput(job, "operator_market_share", "operator", "market_share");

        FileInputFormat.addInputPath(job, new Path("data/phone.log"));
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}

package com.yopai.mrmysql;

import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.mapred.lib.db.DBWritable;

import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;

/**
 * @Description
 */
public class DBOutputWritable implements Writable, DBWritable {
    private String operator;
    private double market_share;

    public DBOutputWritable() {
    }

    public DBOutputWritable(String operator, double market_share) {
        this.operator = operator;
        this.market_share = market_share;
    }

    @Override
    public void readFields(DataInput in) throws IOException {
        operator = in.readUTF();
        market_share = in.readDouble();
    }

    @Override
    public void write(DataOutput out) throws IOException, IOException {
        out.writeUTF(operator);
        out.writeDouble(market_share);
    }

    @Override
    public void readFields(ResultSet resultSet) throws SQLException {
        // 不需要實(shí)現(xiàn)此方法，因?yàn)槲覀冎粫?huì)寫(xiě)入數(shù)據(jù)到數(shù)據(jù)庫(kù)
    }

    @Override
    public void write(PreparedStatement preparedStatement) throws SQLException {
        preparedStatement.setString(1, operator);
        preparedStatement.setDouble(2, market_share);
    }
}

運(yùn)行結(jié)果如下
【Hadoop綜合實(shí)踐】手機(jī)賣場(chǎng)大數(shù)據(jù)綜合項(xiàng)目分析

可視化操作

package com.yopai.draw;

/**
 * @Description
 */
import java.awt.*;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;
import javax.swing.JFrame;
import org.jfree.chart.ChartFactory;
import org.jfree.chart.ChartPanel;
import org.jfree.chart.JFreeChart;
import org.jfree.chart.plot.PiePlot;
import org.jfree.data.general.DefaultPieDataset;

public class PieChartExample extends JFrame {

    public PieChartExample() {
        // 從數(shù)據(jù)庫(kù)獲取數(shù)據(jù)
        DefaultPieDataset dataset = new DefaultPieDataset();
        try {
            String dbUrl = "jdbc:mysql://localhost:3306/blog";
            String dbUsername = "root";
            String dbPassword = "Admin2022!";
            Connection connection = DriverManager.getConnection(dbUrl, dbUsername, dbPassword);
            Statement statement = connection.createStatement();
            ResultSet resultSet = statement.executeQuery("SELECT operator, market_share FROM operator_market_share");
            while (resultSet.next()) {
                String operator = resultSet.getString("operator");
                double marketShare = resultSet.getDouble("market_share");
                dataset.setValue(operator, marketShare);
            }
        } catch (Exception e) {
            e.printStackTrace();
        }

        // 創(chuàng)建餅圖
        JFreeChart pieChart = ChartFactory.createPieChart(
                "運(yùn)營(yíng)商市場(chǎng)占比",   // 圖表標(biāo)題
                dataset,          // 數(shù)據(jù)集
                true,             // 是否顯示圖例
                true,             // 是否生成工具提示
                false             // 是否生成URL鏈接
        );
        // 設(shè)置字體以顯示中文
        Font font = new Font("宋體", Font.PLAIN, 12);
        pieChart.getTitle().setFont(font);
        pieChart.getLegend().setItemFont(font);
        PiePlot plot = (PiePlot) pieChart.getPlot();
        plot.setLabelFont(font);
        // 添加餅圖到面板并顯示
        ChartPanel chartPanel = new ChartPanel(pieChart);
        setContentPane(chartPanel);
    }

    public static void main(String[] args) {
        PieChartExample pieChartExample = new PieChartExample();
        pieChartExample.setSize(600, 600);
        pieChartExample.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        pieChartExample.setVisible(true);
    }
}

結(jié)果如下
【Hadoop綜合實(shí)踐】手機(jī)賣場(chǎng)大數(shù)據(jù)綜合項(xiàng)目分析
本篇文章到這里結(jié)束需要注意的是每個(gè)人的環(huán)境不用調(diào)用的API會(huì)有所差異。文章來(lái)源地址http://www.zghlxwxcb.cn/news/detail-476897.html

到了這里，關(guān)于【Hadoop綜合實(shí)踐】手機(jī)賣場(chǎng)大數(shù)據(jù)綜合項(xiàng)目分析的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！

本文來(lái)自互聯(lián)網(wǎng)用戶投稿，該文觀點(diǎn)僅代表作者本人，不代表本站立場(chǎng)。本站僅提供信息存儲(chǔ)空間服務(wù)，不擁有所有權(quán)，不承擔(dān)相關(guān)法律責(zé)任。如若轉(zhuǎn)載，請(qǐng)注明出處：如若內(nèi)容造成侵權(quán)/違法違規(guī)/事實(shí)不符，請(qǐng)點(diǎn)擊違法舉報(bào)進(jìn)行投訴反饋，一經(jīng)查實(shí)，立即刪除！

分享到：

領(lǐng)支付寶紅包贊助服務(wù)器費(fèi)用

網(wǎng)約車大數(shù)據(jù)綜合項(xiàng)目——數(shù)據(jù)分析Hive
2024年02月02日
瀏覽(83)
大數(shù)據(jù)綜合項(xiàng)目--網(wǎng)站流量日志數(shù)據(jù)分析系統(tǒng)(詳細(xì)步驟和代碼)
提示：這里簡(jiǎn)述我使用的版本情況： ubuntu16.04 hbase1.1.5 hive1.2.1 sqoop1.4.6 flume1.7.0 項(xiàng)目所使用的參考文檔和代碼資源和部分?jǐn)?shù)據(jù) 網(wǎng)盤(pán)鏈接：鏈接：https://pan.baidu.com/s/1TIKHMBmEFPiOv48pxBKn2w 提取碼：0830 為更好的理解項(xiàng)目架構(gòu)，對(duì)項(xiàng)目使用的一些服務(wù)補(bǔ)充一些基本概述：什么是Sqoop
2023年04月08日
瀏覽(20)
【hadoop】汽車銷售數(shù)據(jù)統(tǒng)計(jì)分析項(xiàng)目（部分）
來(lái)源：《hadoop大數(shù)據(jù)開(kāi)發(fā)實(shí)戰(zhàn)》實(shí)驗(yàn)一：統(tǒng)計(jì)乘用車輛和商用車輛的數(shù)量和銷售額分布設(shè)計(jì)思路：首先，寫(xiě)一個(gè)Mapper來(lái)映射輸出所有乘用車輛（feiyingyun）和商用車輛（yingyun）的記錄。然后，寫(xiě)一個(gè)reduce統(tǒng)計(jì)出乘用車輛和商用車輛各自的數(shù)量,寫(xiě)一個(gè)map的映射集合中，其中
2024年02月09日
瀏覽(97)
加速大規(guī)模數(shù)據(jù)處理和多維分析：基于Lucene和Hadoop的開(kāi)源項(xiàng)目
大數(shù)據(jù)時(shí)代帶來(lái)了處理和分析海量數(shù)據(jù)的挑戰(zhàn)，我很高興向大家介紹我的個(gè)人開(kāi)源項(xiàng)目：Lucene-Hadoop。這個(gè)項(xiàng)目基于Lucene和Hadoop，旨在提供高效的數(shù)據(jù)存儲(chǔ)和查詢引擎，加速大規(guī)模數(shù)據(jù)處理和多維分析。項(xiàng)目介紹 https://github.com/arlixu/lucene-hadoop Lucene-Hadoop利用Lucene和Hadoop的強(qiáng)大
2024年02月08日
瀏覽(25)
《軟件需求分析與設(shè)計(jì)》課程綜合實(shí)踐——需求規(guī)格說(shuō)明書(shū)
東莞理工學(xué)院《軟件需求分析與設(shè)計(jì)》課程綜合實(shí)踐項(xiàng)目名稱：《XX培訓(xùn)機(jī)構(gòu)教學(xué)管理系統(tǒng)》需求規(guī)格說(shuō)明書(shū) 學(xué)院：計(jì)算機(jī)與網(wǎng)絡(luò)安全學(xué)院班級(jí)：小組成員： 1 引言 1.1 編寫(xiě)目的 1.2 背景 1.3 定義 1.4 參考資料 2 任務(wù)概述 2.1 目標(biāo) 2.2 用戶的特點(diǎn) 2.3 假定和約束 3 需求規(guī)定 3
2024年02月09日
瀏覽(21)
猿創(chuàng)征文｜Hadoop大數(shù)據(jù)技術(shù)綜合實(shí)驗(yàn)
當(dāng)前互聯(lián)網(wǎng)應(yīng)用中，萬(wàn)維網(wǎng)（World Wide Web）應(yīng)用占據(jù)了絕大部分的份額。萬(wàn)維網(wǎng)應(yīng)用對(duì)外提供服務(wù)需要架設(shè)Web服務(wù)器軟件。典型的Web服務(wù)器軟件有Apache、Nginx等。Web服務(wù)器軟件在運(yùn)行過(guò)程中會(huì)寫(xiě)入各種日志到磁盤(pán)文件中。例如，Apache Web服務(wù)器軟件運(yùn)行過(guò)程中，會(huì)產(chǎn)生access.log文
2024年02月03日
瀏覽(31)
數(shù)字IC實(shí)踐項(xiàng)目（2）——高速SDRAM控制器的設(shè)計(jì)與綜合（入門(mén)級(jí)工程項(xiàng)目）
這個(gè)實(shí)踐項(xiàng)目來(lái)源于研究生電子設(shè)計(jì)競(jìng)賽，在涉及到視頻圖像處理時(shí)需要用到DRAM存儲(chǔ)數(shù)據(jù) ;整個(gè)項(xiàng)目過(guò)程中先后學(xué)習(xí)了小梅哥（AC620開(kāi)發(fā)板資料）、開(kāi)源騷客SDRAM控制器、正點(diǎn)原子FPGA教程、野火FPGA開(kāi)發(fā)教程等網(wǎng)絡(luò)資料。在此對(duì)上述提供學(xué)習(xí)資料的前輩表示真誠(chéng)的感謝。
2024年02月13日
瀏覽(35)
【Spark+Hadoop+Hive+MySQL+Presto+SpringBoot+Echarts】基于大數(shù)據(jù)技術(shù)的用戶日志數(shù)據(jù)分析及可視化平臺(tái)搭建項(xiàng)目
點(diǎn)我獲取項(xiàng)目數(shù)據(jù)集及代碼隨著我國(guó)科學(xué)技術(shù)水平的不斷發(fā)展，計(jì)算機(jī)網(wǎng)絡(luò)技術(shù)的廣泛應(yīng)用，我國(guó)已經(jīng)步入了大數(shù)據(jù)時(shí)代。在大數(shù)據(jù)背景下，各種繁雜的數(shù)據(jù)層出不窮，一時(shí)難以掌握其基本特征及一般規(guī)律,這也給企業(yè)的運(yùn)營(yíng)數(shù)據(jù)分析工作增添了不小的難度。在大數(shù)據(jù)的背景
2024年02月10日
瀏覽(28)
基于C#和MATLAB對(duì)手機(jī)錄音音頻數(shù)據(jù)分析處理系統(tǒng) 畢業(yè)論文+項(xiàng)目源碼
摘要當(dāng)今科學(xué)技術(shù)發(fā)展迅猛，知識(shí)爆炸信息量的急劇增加不僅僅豐富了我們的現(xiàn)實(shí)生活，也對(duì)我們的信息處理技術(shù)提出了新的要求。音頻信號(hào)在這信息洪流中占據(jù)著不可或缺的重要地位，諸如語(yǔ)音聊天，音頻取證等在我們的生活中發(fā)揮著愈來(lái)愈重要的作用，故而對(duì)于音頻的特
2024年01月19日
瀏覽(26)
【項(xiàng)目實(shí)戰(zhàn)】基于Hadoop大數(shù)據(jù)電商平臺(tái)用戶行為分析與可視化系統(tǒng)Hive、Spark計(jì)算機(jī)程序開(kāi)發(fā)
注意：該項(xiàng)目只展示部分功能，如需了解，評(píng)論區(qū)咨詢即可。在當(dāng)今數(shù)字化時(shí)代，電商行業(yè)成為全球商業(yè)生態(tài)系統(tǒng)的關(guān)鍵組成部分，電商平臺(tái)已經(jīng)深入各行各業(yè)，影響了人們的購(gòu)物方式和消費(fèi)習(xí)慣。隨著互聯(lián)網(wǎng)技術(shù)的不斷發(fā)展，電商平臺(tái)產(chǎn)生了大量的用戶數(shù)據(jù)，包括點(diǎn)擊、購(gòu)
2024年02月04日
瀏覽(111)

<style id="u5ddj"><nav id="u5ddj"><tt id="u5ddj"></tt></nav></style>