When attempting to read a CSV file from another application I get this error. when reading the first line itself.
java.lang.IllegalArgumentException: Mapping for season not found, expected one of [AddonCommissionAmount, AddonDiscountAmount, AddonFeeAmount, AddonItemAmount, AddonNetAmount, BalanceAmount, BookingPolicy, Brand, CabinCode, CabinNum, Campaign, CancelConfirmDate, Comments, ConfirmTime, CreatedOn1, GroupId, GroupName, KayakCommissionAmount, KayakDiscountAmount, KayakFeeAmount, KayakItemAmount, KayakNetAmount, MarketMngmt, OptionDate, Payment1Amount, Payment1Balance, Payment1Date, Payment1PaidAmount, Payment1PayMethod, Payment2Amount, Payment2Balance, Payment2Date, Payment2PaidAmount, Payment2PayMethod, Payment3Amount, Payment3Balance, Payment3Date, Payment3PaidAmount, Payment3PayMethod, Payment4Amount, Payment4Balance, Payment4Date, Payment4PaidAmount, Payment4PayMethod, PrimaryCommissionAmount, PrimaryDiscountAmount, PrimaryFeeAmount, PrimaryItemAmount, PrimaryNetAmount, PrimaryPartyCountry, PrimaryPartyEmail, PrimaryPartyName, ProductCode, RezAgent, RezDiscardOption, RezId, RezInvoice, RezPartyCnt, RezStatusName, SeatingCommissionAmount, SeatingDiscountAmount, SeatingFeeAmount, SeatingItemAmount, SeatingNetAmount, Softland, StartDate, TotalAmount, TotalPaidAmount, TravelAgentCompany1, TravelAgentCountry1, TravelAgentEmail1, TravelAgentName1, Venue, Season] at org.apache.commons.csv.CSVRecord.get(CSVRecord.java:121) at com.antarctica21.backend.rpa.SaveRpaReportData.runCronJob(SaveRpaReportData.java:73)The First Line of the CSV is printed in the log - Initial (Error)
[comment='null', recordNumber=1, values=[2022-23, M-CA_2022-23, 2022-23 AIR-CRUISE, 12/1/2022, MExplorer, 1045856, , , Cancelled, , , , CHR_Tripolers_M_2022-23, 88930, 2/17/2021, , 2/17/2021, 9/6/2021, Tripoler Charter 1, 0, 307, Single Cabin, Beijing Travel Co., Ltd, Yan Xu, , China, Asia, , , Emilia, , 0.00, , , 0.00, 0.00, , , , , 0.00, , , , , 0.00, 0.00, , , , 0.00, 0.00, 1/13/2021, 15,000.00, Wire Transfer, 15,000.00, 0.00, 3/15/2021, 40,000.00, Wire Transfer, 40,000.00, 0.00, 5/15/2021, 40,000.00, Wire Transfer, 40,000.00, 0.00, 6/13/2022, -95,000.00, Transfer, -95,000.00, 0.00, 0.00,0.00]]The First Line of the CSV printed in the log - Once resaved via sublime (No error)
[comment='null', recordNumber=1, values=[2022-23, M-CA_2022-23, 2022-23 AIR-CRUISE, 12/1/2022, MExplorer, 1045856, , , Cancelled, , , , CHR_Tripolers_MAG07_2022-23, 88930, 2/17/2021, , 2/17/2021, 9/6/2021, Tripoler Charter 1, 0, 307, Single Cabin, Beijing Travel Co., Ltd, Yan Xu, , China, Asia, , , Emilia, , 0.00, , , 0.00, 0.00, , , , , 0.00, , , , , 0.00, 0.00, , , , 0.00, 0.00, 1/13/2021, 15,000.00, Wire Transfer, 15,000.00, 0.00, 3/15/2021, 40,000.00, Wire Transfer, 40,000.00, 0.00, 5/15/2021, 40,000.00, Wire Transfer, 40,000.00, 0.00, 6/13/2022, -95,000.00, Transfer, -$95,000.00, $0.00, $0.00, $0.00]]But when the same CSV file contents are copied into a new tab in the sublime text editor and saved as a new file the CSV becomes parsable and code works file. Did used many methods to look for special characters or formatting but nothing was indicated via the methods used.
what is the process that can be used to clean the CSV file programmatically to make it parsable alike when saved via the sublime text editor? (The same process with other applications did not work.) what methods should be devised to find what causes these errors and how it is removed? After trying out so many alternative methods I am not clear on how this can be achieved.
Related sample Java class
import com.antarctica21.backend.sales.entity.Sale;import com.antarctica21.backend.sales.repository.SaleRepository;import com.antarctica21.backend.util.EMailSender;import lombok.RequiredArgsConstructor;import lombok.extern.slf4j.Slf4j;import org.apache.commons.csv.CSVFormat;import org.apache.commons.csv.CSVParser;import org.apache.commons.csv.CSVRecord;import org.springframework.core.env.Environment;import org.springframework.scheduling.annotation.Scheduled;import org.springframework.stereotype.Component;import org.springframework.transaction.annotation.Transactional;import java.io.FileReader;import java.io.Reader;import java.math.BigDecimal;import java.text.NumberFormat;import java.text.ParseException;import java.text.SimpleDateFormat;import java.time.LocalDateTime;import java.time.format.DateTimeFormatter;import java.util.ArrayList;import java.util.Date;@Slf4j@Component@RequiredArgsConstructorpublic class SaveReportData { private final Environment env; private final SaleRepository salesRepository; private final EMailSender eMailSender; private Boolean jobEnabled = true; @Transactional(rollbackFor = Throwable.class). @Scheduled(cron = "*/20 * * * * *") // Scheduled for every 10 seconds. public void runCronJob() { if (!jobEnabled){ return; } String csvFilePath = "Sales Report.csv"; try (Reader reader = new FileReader(csvFilePath); CSVParser csvParser = new CSVParser(reader, CSVFormat.DEFAULT.withFirstRecordAsHeader().withIgnoreHeaderCase())) { ArrayList<Sale> csvRecords = new ArrayList<>(); for (CSVRecord csvRecord : csvParser) { System.out.println("csvRecord: " + csvRecord); // Fails here Sale sale = new Sale(); sale.setSeason(csvRecord.get("season")); sale.setTripCode(csvRecord.get("productCode")); sale.setDepartureDate(convertStringToDate(csvRecord.get("startDate"))); sale.setComments(csvRecord.get("comments")); //Fields omitted for clarity csvRecords.add(sale); } // Save all records in the csvRecords array. salesRepository.saveAll(csvRecords); } catch (Throwable e) { log.error("An error occurred", e); eMailSender.sendErrorReportEmail(env.getProperty("report.email"), e.getMessage(), getCurrentDateAndTime()); jobEnabled = false; } }